数据科学 (Data science)

接VBA, Python, R咨询

Mengkelyu — Sat, 26 Mar 2022 12:52:10 GMT

也欢迎大家加我的微信MengkeLyuActuary，我会把大家加到我的代码群

SnakeBytes 2/25: Corbin Barrels Slade

ArizonaDiamond — Fri, 25 Apr 2025 02:12:18 GMT

Diamondbacks Information Lovullo states Opening Working day rookie TBD even with providing clues through Nick Piecoro [AZ CentralThat is towards say, if Gallen remained upon a 5-working day timetable the take it easy of the spring, he would line up towards consider the ball upon Opening Working day with a single more working day of take it easy. As for Burnes, he strains up completely towards pitch Working day 2 from the Chicago Cubs upon March Gallen Throws A person Inning, Diamondbacks Order Initially Cactus League Get through Jack Sommers [SIGallen mentioned he felt fantastic. "When compared toward how I felt inside of my past stay [batting educate, I'll get it. I felt a great deal superior https://www.diamondbacksgate.com/collections/jose-herrera-jersey
. I felt which include nearly anything experienced its condition for the utmost component. The changeup that basically style of was flat upon me. It was the property operate, still anything at all else felt attractive powerful. "Gallen chips off the rust, K's 2 within spring debut through Steve Gilbert [Thus much this spring, the co-aces of the workforce contain not experienced a good deal of year jointly, still Gallen stated that would be substitute appear the regular monthly period. "Our schedules consist of been a tiny substitute," Gallen mentioned. "I believe he will work upon a minor former program than I do inside words and phrases of he turns into inside of and becomes the things finished. I'm further more of a gentleman who basically sort of hangs in the vicinity of listed here and does my things afterwards within just the afternoon. Thus we style of miss out on each individual other for breakfast and things which include that, nevertheless we've been within just classes for [drills. "Diamonbacks Reliever Talks Success, Clubhouse Management by way of Alex D'Agostino [SI"We include a good deal of gentlemen inside this clubhouse towards a main and I feel a management see that sort of continue to keep other men within just line. Hence even if an individual functions out of line, I imagine we can variety of maintain that back again within and chat relating to it," Ginkel Carroll prospects Diamondbacks in direction of very first spring gain as a result of Michael McDermott [Venom StrikesStar outfielder Corbin Carroll experienced a effective working day at the plate, homering and drawing a stroll that sooner or later grew to become the work that created the 's residence function arrived versus a previous Diamondback Slade Cecconi, who was delivered toward Cleveland in just the Josh Naylor excess of the LeagueTigers Indication Andrew Chafin Towards Very little League Bundle as a result of Darragh MacDonald [MLB Exchange RumorsIf Chafin appears including his aged self in just camp, he may perhaps potentially consider a task as the instant lefty inside the bullpen along with Holton. If not, he'll consist of options towards physical appearance in other places. Below the present-day collective bargaining arrangement, Write-up XX totally free brokers attain certain choose-out opportunities upon tiny league specials if they are signed a lot more than 10 times earlier in the direction of Opening Working day. An Short article XX no cost consultant is a single with 6 many years of assistance who performed the final time upon a 40-person roster or 60-working day wounded record. All those choose-out opportunities are 5 times preceding toward Opening Working day, Could possibly 1st and June Would a Vladito Agreement Appear to be Which include through Dan Szymborski [FanGraphs As a result at present the Blue Jays track down on their own at a crossroads. Individually, I believe that their failure toward signal Guerrero in advance of his deadline really should be taken care of as the lethal blow in the direction of the organizationnot specifically for the up coming several yrs, nevertheless for this one particular as very well. A large amount of factors would require toward shift straight for Toronto in direction of transform a Guerrero-considerably less roster into a severe AL East risk if he leaves as a cost-free consultant. Thinking about this, the Jays may perhaps be far better off investing Guerrerothey would virtually totally purchase an ideal return package deal for himto retool for yet another operate 2 or a few yrs towards at present, even When that would all still formally consider them out of rivalry for the 2025 marketing campaign https://www.diamondbacksgate.com/collections/luis-gonzalez-jersey. No matter what comes about, simply because the Blue Jays didn't grow the facial area of their franchise, this is shaping up in the direction of be their greatest consequential period due to the fact the cements enjoy for Philly with Phonetic tattoo by way of Todd Zolecky [MLB "I've been searching it for a When," Harper explained. "In a natural way, the Phanatic and Philly necessarily mean a great deal. Yet after I consider around the Phanatic, it's him upon an ATV. I principle it was Very neat. "Cleveland Guardians proprietor Lawrence Dolan dies at 94 [ESPN "Mr. Dolan expended his finish everyday living within Much better Cleveland and afflicted our area upon consequently several stages," Bob DiBiasio, Guardians senior vice president of community affairs, stated within a assertion. "Against his assistance in the direction of our region as a Initially Lieutenant within just the United Claims Marines, his a lot of philanthropic functions of kindness, vocation in just regulation, business enterprise, education and learning, and sporting activities, a lot of benefited against his engagement, have an affect on, and pastime. Specially inside of the globe of sporting activities. "Speculate of the DayPollIf Lovullo delivers Gallen the Opening Working day commence, it will be. https://www.diamondbacksgate.com/collections/jake-mccarthy-jersey. This poll is shut 37% the specifically conclusion8% a miscalculation54% a nothingburger 61 votes totalVote Previously

[小技巧] 如何快速批量复制粘贴并重命名文件(Windows)

Mengkelyu — Mon, 02 Jan 2023 12:19:53 GMT

如果在工作中需要批量复制粘贴并重命名文件，可以使用这个小技巧

首先，打开Windows的PowerShell

在Powershell里面运行

copy "需要复制的文件名地址" "目标文件名地址"

例如

copy "I:\my file.xlsx" "I:\my file (Copied).xlsx"

如果要批量运行，可以先在Excel中用公式写好，然后复制粘贴到Powershell里面运行。

例子如下，公式已经用红色标出

[Youtube学习笔记]Linux的50个实用指令

Mengkelyu — Sat, 10 Dec 2022 10:22:28 GMT

cat file #print all contents of a file 输出一个文件中的全部内容 cat file1 file2 #print all contents of file1 and file2 输出文件1和文件2的全部内容 cat -n file1 #-n is to print out the line numbers 加入参数n，可以同时输出内容和行数 less # it shows the content stored inside a file, in a nice and interactive UI； cat的漂亮形式 echo #it prints to the output the argument passed to it 这个命令等于print wc #print number of lines; number of words; number of bits 输出基本信息，（多少行，多少数字，多少bit） | #give the output of the first argument to the second argument 这个符号可以把符号前面的Output传递给符号后面的命令作为Input ls -l | wc #word count for the ls -l 比如这个例子，ls -l 输出了当前文件夹下所有文件的名字，然后这个文本传递给了wc输出文本的基本信息 sort #sort the document 给文件排序 sort -n file #sort everything numerically 按照数字顺序给文件排序 uniq #report or omit repeated lines uniq -d #only report duplicated values uniq -u #only report non-duplicated values uniq -C #count how many times each value is duplicated *.txt #match all documents ending with txt ? #match one character {a,b,c}.txt #output a.txt, b.txt, c.txt Day{1..365} #output Day1 Day2 Day3 .... Day 365 diff #compare the contents of two files diff -v file1 file2 #compare two files line by line find find . -name #*.txt# find . -iname #*.Txt# #i for case incensitive find . -type d #only search for directory find . -type f #only search for files find . -name #E*# -or -name #*E# find . -type f exec cat {} \ #exec cat to every file. {} is placeholder for each found result grep #help search inside files grep grep -r #recursively found a string in all texts in a folder. -ri can make it case incensitive du #find the size of directories or files df #show how many space left history #history of all commands run. Combined with grep, it is possible to obtain the command with certain character ps #view the process running on the computer top #display top 10 process kill #shut a process kill kill -9 #brutal kill kill -15 #gental kill killall #perform multiple kills jobs #print out commands fg #run some command on the foreground bg #run some command on the background & #run the command on the backgound jobs #see those commands gzip #compress a file gzip -k #compress a file while keeping the original file gzip -d #decompress a file gunzip #decompress a file tar #used to create an archive, grouping multiple files in a single file tar -cf archive.tar file1 file2 #the c option standas for create, the f option is used to write to file the archive tar -xf #extract to current folder tar -czf archive.tar.gz file1 file2 #creating a tar achive and then running gzip on it nano #a friendly editor alias = ''# Give some alias to a long command # difference of "" and '', "" allows variable in a string echo "home directory \$PWD" # 这里\$PWD被当作成了变量或参数 echo 'home directory \$PWD' # 这里\$PWD没有被当作成变量或参数 xargs # turn the output of first argument to the input of second arguement' find x -size +1M | xargs ls -lh ln # it is used to create links. Links are effectively a pointer to another file # Two types of links: hard links and soft links ln # create hard link to file1 ln -s # create soft link # soft link will be removed when original file is removed. But hard link will not be removed

[小技巧] 如何快速Set Up文件夹（适用于Windows)

Mengkelyu — Wed, 11 May 2022 06:30:44 GMT

工作中在新开Project的时候，有时候会需要按照一定规律放置一些文件夹。一个一个新建文件夹太麻烦了，有没有什么快速的办法呢？

有的！那就是CMD。

假设在这个文件夹下面开始新的Project

要建立下面几个文件夹

1 Product Spec
2 BE&MP
3 Pricing Tool
4 Pricing Note
5 Term Sheet
6 Treaty
7 UW
8 Disease Definition
在这里输入PowerShell

点击回车，出现下面的界面

输入

md '1 Product Spec', '2 BE&MP', '3 Pricing Tool' ,'4 Pricing Note', '5 Term Sheet', '6 Treaty', '7 UW', '8 Disease Definition'

点击回车运行，文件夹就建好了

免费开源Programming书籍集合

Mengkelyu — Sat, 05 Mar 2022 17:21:38 GMT

GIthub上有这样一个项目收录了很多免费的编程教材。推荐给大家。

https://github.com/EbookFoundation/free-programming-books/blob/main/books/free-programming-books-langs.md#bash

经典教材：Introduction to Data Mining_2ed_Pang Ning Tan

Howie Jie — Sat, 26 Feb 2022 14:38:04 GMT

经典教材：Introduction to Data Mining_2ed_Pang Ning Tan

Introduction to Data Mining_2ed_Pang Ning Tan.pdf

精算用的软件集合

Mengkelyu — Fri, 18 Jun 2021 02:55:16 GMT

寿险

Axis：价格便宜，灵活性差。中宏，合众，慕再，英大，长生，光大永明等（2021.6.17）
Prophet：价格贵，灵活性高。用的较多

财险

Sas：商业软件，较贵
Python

评估：

Excel

一行代码EDA：Python和R中可以用一行代码进行数据探索性分析的包介绍和对比

Mengkelyu — Sat, 15 May 2021 11:12:41 GMT

AutoViz 照例，先安装 python -m venv au-ven au-ven\Scripts\activate.ps1 pip install autoviz 然后报了错，首先是第一个错是 ModuleNotFoundError: No module named 'scipy.special.cython_special'上网查了一下，原来是scipy版本不兼容，于是乎把scipy降到了1.4.1版本 python -m pip install scipy==1.4.1 之后发现，原来必须要用jupyter来跑这个包，因为它无法像Sweetiviz等包生成html版本的report。 from autoviz.AutoViz_Class import AutoViz_Class # 数据集来源：https://github.com/hmix13/AutoViz AV = AutoViz_Class() df = AV.AutoViz('car_design.csv') Jupyter Lab -> There you go! 说实话，没想到这个包震惊到了我，它的图实在太好康了

从动态网页中爬Request的经验

Mengkelyu — Wed, 14 Apr 2021 13:23:24 GMT

由于各种原因，不方便透露是哪两个网站。但是可以和大家分享一下一些心得。第一个网站中，关键的突破点是通过浏览器的网络监控（Chrome点击F12，并选择Network如下图），点击Ctrl+F，输入想要爬取信息的关键词（我这里是“平安”），成功检索到了网站提取信息的Url。第二个网站就更有意思了，我发现这个网页是首先把所有的数据提取出来后，分页完全是前端进行的。而数据变量还存储在浏览器中。我找到数据变量"List"后，在浏览器console运行了如下的js代码，这是一个用于储存浏览器中变量的函数。 (function(console){ console.save = function(data, filename){ if(!data) { console.error('Console.save: No data') return; } if(!filename) filename = 'console.json' if(typeof data === "object"){ data = JSON.stringify(data, undefined, 4) } var blob = new Blob([data], {type: 'text/json'}), e = document.createEvent('MouseEvents'), a = document.createElement('a') a.download = filename a.href = window.URL.createObjectURL(blob) a.dataset.downloadurl = ['text/json', a.download, a.href].join(':') e.initMouseEvent('click', true, false, window, 0, 0, 0, 0, 0, false, false, false, false, 0, null) a.dispatchEvent(e) } })(console) 然后运行 console.save(List, "List.txt") 变量List就存储到List.txt这个文档中了。基本上整个过程是无代码爬虫。

如何用Bokeh和ggplot制作Grouped and Stacked Column Chart

Mengkelyu — Thu, 24 Sep 2020 13:38:58 GMT

Bokeh的效果图：

三种Ensemble的办法（Bagging, boosting, stacking)

Mengkelyu — Tue, 21 Jul 2020 07:15:52 GMT

根据我实际的测试，Bagging后的结果确实比单次的结果好很多。大概能提升AUC 0.01左右

缺失值处理的方法

Mengkelyu — Sat, 30 May 2020 15:25:22 GMT

决定处理方法的核心问题：数据缺失的原因是什么？手里的样本是大样本还是小样本？

删除记录

删除缺失值对应的行或列

行或列缺失占比较大时考虑这个选项
某一特征缺失20%以上，需要考虑删除整列
无法使用合适的方法进行缺失值插补

插补

以均值/众数/中位数等填充缺失值

注意：解决数据右偏问题后（例如用log(1+x)处理某特征），再填充。
内涵假设：缺失数据的分布与未缺失数据分布相同。当缺失数据为随机缺失时有效

聚类填充：根据其它特征进行聚类，然后填充最有可能的值

最近邻插补：在样本中找到与缺失样本最接近的样本的该属性值进行插补
核心假设：缺失数据的分布与与其同类的数据相同。
举例：苹果的糖分含量用苹果的数据估计，香蕉的糖分含量用香蕉的数据估计

根据实际意义以指定值填充

插值法(利用已知点建立合适的插值函数$f(x)$，未知值由对应点$x_i$求出的函数值$f(x_i)$近似代替)

拉格朗日插值法
牛顿插值法
其他插值法：Hermite插值、分段插值、样条插值法等

import pandas as pd
from scipy.interpolate import lagrange

#自定义列向量拉格朗日插值函数
#s 为列向量，n为被插值的位置，k为取前后的数据个数，默认为5
def ployinterp_column(s, n, k=5):
    y = s[list(range(n-k, n))+list(range(n+1, n+1+k))]#取数
    y = y[y.notnull()] #剔除空值
    return lagrange(y.index, list(y))(n) # 插值并返回插值结果

#逐个元素判断是否需要插值
for i in data.columns:
    for j in range(len(data)):
        if (data[i].isnull())[j]: #如果为空即插值
            data[i][j] = ployinterp_column(data[i], j)

填充了缺失值后, 再加一列indicator

内涵假设：缺失数据是有某一特征的数据，所以这部分数据的行为和其它数据不一样

直接把缺失值所在的列当做y, 缺失值所在的行当做测试集，其它行当做训练集

需要足够有效样本

不处理

不处理可能很多模型跑不通，还不知道为啥会有不处理选项