Ethereum GPU Mining on Linux How-To

https://www.meebey.net/posts/ethereum_gpu_mining_on_linux_howto/

 

TL;DR

Install/use Debian 8 or Ubuntu 16.04 then execute:

sudo apt-get install software-properties-common
sudo add-apt-repository ppa:ethereum/ethereum
sudo sed 's/jessie/vivid/' -i /etc/apt/sources.list.d/ethereum-ethereum-*.list
sudo apt-get update
sudo apt-get install ethereum ethminer
geth account new
# copy long character sequence within {}, that is your <YOUR_WALLET_ADDRESS>
# if you lose the passphrase, you lose your coins!
sudo apt-get install linux-headers-amd64 build-essential
wget https://us.download.nvidia.com/XFree86/Linux-x86_64/384.90/NVIDIA-Linux-x86_64-384.90.run
chmod +x NVIDIA-Linux-x86_64-384.90.run
sudo ./NVIDIA-Linux-x86_64-384.90.run
ethminer -G -F http://yolo.ethclassic.faith:9999/0x<YOUR_WALLET_ADDRESS> --farm-recheck 200
echo done

My Attention Span is > 60 seconds

Ethereum is a crypto currency similar to Bitcoin as it is based on the blockchain technology. Ethereum is not yet another Bitcoin clone though, since it has an additional feature called Smart Contracts that makes it unique and very promising. I am not going into details how Ethereum works, you can get that into great detail on the Internet. This post is about Ethereum mining. Mining is how crypto coins are created. You need to spent computing time to get coins out. At the beginning CPU mining was sufficient, but as the Ethereum network difficulty has increased you need to use GPUs as they can calculate at a much higher hashrate than a general purpose CPU can do.

About 2 months ago I bought a new gaming rig, with a Nvidia GTX 1070 so I can experience virtual-reality gaming with a HTC Vive at a great framerate. As it turns out modern graphics cards are very good at hashing so I gave it a spin.

Initially I did this mining setup with Windows 10, as that is the operating system on my gaming rig. If you want to do Ethereum mining using your GPU, then you really want to use Linux. On Windows the GTX 1070 produced a hashrate of 6 MH/s (megahashes per second) while the same hardware does 25 MH/s on Linux. The hashrate multiplied by 4 by using Linux instead of Windows. Sounds good? Keep reading and follow this guide.

You have to pick a Linux distro to use for mining. As I am a Debian developer, all my systems run Debian, which is what I am also using for this guide. The same procedure can be done for Ubuntu as it is similar enough. For other distros you have to substitute the steps yourself. So I assume you already have Debian 8 or Ubuntu 16.04 installed on your system.

Install Ethereum Software

First we need the geth tool which is the main Ethereum “client”. Ethereum is really a peer-to-peer network, that means each node is a server and client at the same time. A node that contains the complete blockchain history in a database is called a full node. For this guide you don’t need to run a full node, as mining pools do this for you. We still need geth to create the private key of your Ethereum wallet. Somewhere we have to receive the coins we are mining ;)

Add the Ethereum APT repository using these commands:

sudo apt-get install software-properties-common
sudo add-apt-repository ppa:ethereum/ethereum
sudo apt-get update

On Debian 8 (on Ubuntu you can skip this) you need to replace the repository name with this command:

sudo sed 's/jessie/vivid/' -i /etc/apt/sources.list.d/ethereum-ethereum-*.list
sudo apt-get update

Install ethereum, ethminer and geth:

sudo apt-get install ethereum ethminer geth

Create Ethereum Wallet

A wallet is where coins are “stored”. They are not really stored in the wallet because the wallet is just a private key that nobody has. The balance of that wallet is visible to everyone using the blockchain database. And this is what full nodes do, they contain and distribute the database to all other peers. So this this command to create your first private key for your wallet:

geth account new

Be aware, that this passphrase protects the private key of your wallet. Anyone who has access to that file and knows your passphrase will have full control over your coins. And also do not forget the passphrase, as if you do, you lost all your coins!

The output of “geth account new” shows a long character/number sequence quoted in {}. This is your wallet address and you should write that number down, as if someone wants to send you money, then it is to that address. We will use that for the mining pool later.

Install (proprietary) nvidia driver

For OpenCL or CUDA to work with nvidia graphics cards, like my GTX 1070, you need to install this proprietary driver from nvidia. If you have an older card maybe the opensource drivers will work for you. For the GTX 9xx or 10xx you will need this driver package.

After you have agreed the terms, download the NVIDIA-Linux-x86_64-XXX.XX.run file. But before we can use that installer we need to install some dependencies that installer needs as it will have to compile a Linux kernel module for you. Install the dependencies using this command:

sudo apt-get install linux-headers-amd64 build-essential

Now we can make the installer executable and run it like this:

chmod +x NVIDIA-Linux-x86_64-384.90.run
sudo ./NVIDIA-Linux-x86_64-384.90.run

If that step completed without error, then we should be able to run the mining benchmark!

ethminer -M -G

The -M means “run benchmark” and the -G is for GPU mining. The first time you run it it will create a DAG file and that will takes a while. For me it took about 12 minutes on my GTX 1070. After that is should show a inner mean hashrate. If it says H/s that is hashes per second and KH is kilo (H/1000) and MH is megahashes per second (KH/1000). I had numbers around 25-30 MH/s, but for real mining you will see an average that is a balanced number and not a min/max range.

Pick Ethereum Network

Now it gets serious, you need to decide 2 things. First which Ethereum network you want to mine for and the second is using which pool.

Ethereum has 2 networks, one is called Ethereum One or Core, while the other is called Ethereum Classic. Ethereum has made a hardfork to undo the consequences of a software bug in the DAO. The DAO is a smart contract for a decentralized organization. Because of that bug, a blackhat could use that bug to obtain money from that DAO. The Ethereum developers made a poll and decided that the consequences will be undone. Not everyone agreed and the old network stayed alive and is now called Ethereum Classic short ETC. The hardfork kept its short name ETH.

This is important to understand for mining, because the hashing difficulty has a huge difference between ETH and ETC. As of writing, the hashrate of ETC is at 20% compared to ETH. Thus you need less computing time to get ETC coins and more time to get ETH coins. Differently said, ETC mining is currently more profitable.

Pick a Pool

Hmmmm, I want a swimming pool, thanks! Just kidding… You can mine without a pool, that is called solo mining, but you will get less reward. A mining pool are multiple computers that work on the same block to find a solution quicker than others. The pool has an aggregated hashrate that is higher than other solo miners. Each found block by anyone in this pool will be rewarded to everyone in the pool. The reward of 5 ether currently per block gets split in the same ratio of hashrate each member provides (minus the pool fee). So while you get less for a found block, you still have a steady lower income rate instead of higher with less chance of finding one (in time). Simply said: you have to find a new block faster than the others to receive the reward.

If you want to mine Ethereum Classic (ETC) use one of the pools listed here (at the bottom of the page).

If you want to mine Ethereum One / Core (ETH) use one of the pools listed here.

Run ethminer

The instruction page of the pool website usually says how to start the miner program, but here is an example of the pool that I use (because pony!):

ethminer -G -F http://yolo.ethclassic.faith:9999/0x<YOUR_WALLET_ADDRESS> --farm-recheck 200

Profit

If this guide was helpful for you, you can tip me at ethereum:0x9ec1220d2f2fadd3f0c96e3007daa827bc83fbd6 or simply run the ethminer using my wallet address for a day or two:

ethminer -G -F http://yolo.ethclassic.faith:9999/0x9ec1220d2f2fadd3f0c96e3007daa827bc83fbd6 --farm-recheck 200

If you need a crypto exchange to cash out your mined ether or to buy other crypto currencies, have a look at gatecoin.com When signing up enter WETFAL as referral code and you will get 10 USD after your first deposit.

Happy mining!

Git Pull & Push Using A Custom Port

from http://twincreations.co.uk/git-pull-push-using-custom-port/

 

CentOS 6, Git 1.7

This is a further addition to the article Secure SSH on CentOS 6 And RHEL.

By default, Git will try to connect to a remote host over port 22 (the default SSH port). As a security measure, and as a part of securing your server, we need to be able use Git on an alternate port, view the article for details.

There are a few options available to you, but the only one that works reliably for me is the following method. You should use whatever you are comfortable with. On the local machine, we can tell Git to connect to the remote host on a different port by specifying the port number in our git add declaration.

1
git remote add myserver ssh://8.8.8.8:443/path/to/repo.git

Alternatively, you can directly edit the Git config file.

1
vi .git/config

And then specify the port number and the ssh:// connection method.

1
2
3
4
5
6
7
# a default git remote entry:
[remote “myserver”]
url = root@8.8.8.8:/path/to/repo.git

# using a custom port (other than 22):
[remote “myserver”]
url = ssh://root@8.8.8.8:443/path/to/repo.git

And that’s it. You don’t need to reload anything, it should work straight away. Just use your regular git pull and git push commands.

As I said, there are other methods to get this done. For example, you could just add the following to a ~/.ssh/config file:

1
2
host myserver
port 443

I prefer NOT to use the ~/.ssh/config method because I found it to be unreliable in the long run.

Your choice tho 🙂

Configure tcpdump to work as non-root

https://askubuntu.com/questions/530920/tcpdump-permissions-problem

 

You need to give tcpdump the permission and capability to allow raw packet captures and network interface manipulation.

Add a capture group and add yourself to it:

sudo groupadd pcap
sudo usermod -a -G pcap $USER

Next, change the group of tcpdump and set permissions:

sudo chgrp pcap /usr/sbin/tcpdump
sudo chmod 750 /usr/sbin/tcpdump

Finally, use setcap to give tcpdump the necessary permissions:

sudo setcap cap_net_raw,cap_net_admin=eip /usr/sbin/tcpdump

Be careful, that this will allow everybody from the group pcap to manipulate network interfaces and read raw packets!

天眼查网站的解析记录

最近都在和天眼查的反扒工程师斗智斗勇,记一点关于这个网站的记录。

网址是https://www.tianyancha.com/relation

数据是通过json来交互的。

页面上的请求函数

require([“window”, “jquery”, “util/ejs”, “util/modal”, “util/loading”, “moment”, “layout/main”, “util/UserManage”, “util/common”, “util/liveSearch”, “util/query”, “d3”, “vis”, “toBlob”, “filesave”, “lib/domReady”, “util/vipcontent”], function(t, e, i, s, n, a, o, r, l, c, d, p, u, v, m, h, f)

主要请求函数和对应的变量名。

通过分析可以发现,这里的请求函数的位置是在

d.getShortestPath(I.fromCid, I.fromHid, I.toCid, I.toHid)

主要是通过”util/query”来请求的。请求完数据后再保存到本地。

console的模拟代码

var t = require(‘util/query’).getShortestPath(“651442”, null, “2310299434”, null);

此时t的response就是编号喂651442以及2310299434两个公司的信息。

t.responseJSON
Object { state: “ok”, message: “”, special: “”, vipMessage: “你正在使用<em>VIP权限</em>”, isLogin: 0, data: “{\”p_0\”:{\”nodes\”:[{\”id\”:\”651442\”,\”labels\”:[\”Company\”],\”properties\”:{\”name\”:\”大连万达集团股份有限公司\”}},{\”id\”:\”2310299434\”,\”labels\”:[\”Company\”],\”properties\”:{\”name\”:\”苏州小灵猫信息科技有限公司\”}},{\”id\”:\”4845369\”,\”labels\”:[\”Company\”],\”properties\”:{\”name\”:\”大连万达商业地产股份有限公司\”}},{\”id\”:\”17922482\”,\”labels\”:[\”Company\”],\”properties\”:{\”name\”:\”新疆鼎邦股权投资有限公司\”}},{\”id\”:\”26220133\”,\”labels\”:[\”Company\”],\”properties\”:{\”name\”:\”北京华泰兴农农业科技有限公司\”}},{\”id\”:\”3031679031\”,\”labels\”:[\”Company\”],\”properties\”:{\”name\”:\”南通精慧工业信息咨询中心(有限合伙)\”}},{\”id\”:\”2338840297\”,\”labels\”:[\”Company\”],\”properties\”:{\”name\”:\”苏州千璞精慧投资中心(有限合伙)\”}},{\”id\”:\”2338840347\”,\”labels\”:[\”Company\”],\”properties\”:{\”name\”:\”上海千璞资产管理有限公司\”}},{\”id\”:\”2345120908\”,\”labels\”:[\”Company\”],\”properties\”:{\”name\”:\”苏州卡菲猫投资中心(有限合伙)\”}},{\”id\”:\”2345120907\”,\”labels\”:[\”Company\”],\”properties\”:{\”name\”:\”苏州慧璞投资咨询企业(有限合伙)\”}},{\”id\”:\”2338840298\”,\”labels\”:[\”Company\”],\”properties\”:{\”name\”:\”南通精慧投资中心\”}},{\”id\”:\”2324444865\”,\”labels\”:[\”Company\”],\”properties\”:{\”name\”:\”南通精慧投资中心(有限合伙)\”}}],\”relationships\”:[{\”id\”:\”210231753\”,\”t…” }

不过如果使用这种请求方式来进行数据挖掘的话存在问题,很容易会被甄别出来如图所示,真正的请求还有另外两个交互,该函数就只有这一个函数调用。

可能的话还是要走一走另一条路,

t.panelShowAll = function() {
H(null, null)
}
,
t.panelClick = function(t, e) {
H(null, t)
}

把这个H函数给分析明白了,就可以避免和服务器交互了。H函数的定义如下:

H = function(t, e) {
var i = V(e);
I.nodes = i.nodes,
I.relationships = i.relationships;
var s = U(I.nodes, I.relationships);
I.render(s.nodes, s.relationships, I.startData, I.endData, I.minPathNodesMax)
};

可以看出来H函数的核心是V函数。

V = function(t) {
t = t || “p_0”;
for (var e in I.resultData)
if (e == t)
return I.resultData[e]
}

V函数的核心是调用这里的I中存好的结果,所以接下来的关键就是考察对I进行赋值的那一部分代码,就是回到使用getShortestPath那一部分代码

d.getShortestPath(I.fromCid, I.fromHid, I.toCid, I.toHid).then(function(t) {
if (n.hide(),
t && t.data) {
t.vipMessage && s.showVipMessage(t.vipMessage, 2e3);
var a = {
data: JSON.parse(t.data)
};
a.start = T.from,
a.end = T.to,
I.hideShortPic = !0;
var o = []
, r = 0;
for (var l in a.data)
if (“p_0” != l) {
r++;
var c = {
id: l,
title: “路径” + r,
content: a.data[l].nodes
};
o.push(c)
}
I.panelArr = o;
var d = i.render(“panel”, {
tempArr: o
});
e(“.shortpath_panel”).html(d);
var p = 0
, u = 0;
for (var l in a.data)
p++,
“p_0” != l && (u = a.data[l].nodes.length > u ? a.data[l].nodes.length : u);
1 == p && (u = a.data.p_0.nodes.length),
I.resultData = a.data,
I.startData = a.start,
I.endData = a.end,
I.minPathNodesMax = u,
I.nodes = a.data.p_0.nodes,
I.relationships = a.data.p_0.relationships,
I.render(I.nodes, I.relationships, I.startData, I.endData, I.minPathNodesMax),
U(I.nodes, I.relationships)

这里有几个函数可以分析,一个是U函数,另外一个是I.render。先看render函数:I.render = function(t, i, s, n, a)。从定义以及调用情况可以看出来,这两个函数中的t和i就是我们最关心的两个数据。真麻烦,直接去看原始的就好了。

这是个显示的函数,没啥用。

数据是被requirejs封装起来的,到底该怎么调用啊啊啊啊

Efficient waiting for Ajax call data loading with Selenium WebDriver

Efficient waiting for Ajax call data loading with Selenium WebDriver

 

Post summary: This post is about implementing an efficient mechanism for Selenium WebDriver wait for elements by execution of jQuery code.

Automating single page application with Selenium WebDriver could be sometimes a tricky task. You can get into the trap of timing issues. Although you set explicit waits you still can try to use element that is not yet loaded by the Ajax call. Remember Thread.Sleep() is never an option! You can use very tiny sleeps (100-200ms) in order to wait for initiation of given process, but never use sleep to wait for the end of the process.

Implement Selenium wrapper (Facade)

I good approach I like is to implement your own FindElement method which is basically a wrapper for Selenium’s methods (Facade pattern). With this approach you are hiding unneeded Selenium functionality and have centralised control over location of elements and explicit waits. Locate behaviour of your entire framework is controlled in just one method.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
private static TimeSpan waitForElement = TimeSpan.FromSeconds(10);
public static IWebElement FindElement(By by)
{
    try
    {
        WaitForReady();
        WebDriverWait wait = new WebDriverWait(webDriver, waitForElement);
        return wait.Until(ExpectedConditions.ElementIsVisible(by));
    }
    catch
    {
        return null;
    }
}

Code above is C# one and is implementation of explicit wait with WebDriverWait class from OpenQA.Selenium.Support.UI. You can see an unknown (so far) method WaitForReady(). Note that ElementIsVisible is used instead of ElementExists because element might be on page but yet not ready to work with.

Wait for Ajax call to finish

Initially WaitForReady() was supposed to check that Ajax has finished loading by using jQuery.active property. This is in case jQuery is used in the application under test. If this property is 0 then there are no active Ajax request to server.

1
2
3
4
5
6
private static void WaitForReady()
{
    WebDriverWait wait = new WebDriverWait(webDriver, waitForElement);
    wait.Until(driver => (bool)((IJavaScriptExecutor)driver).
            ExecuteScript("return jQuery.active == 0"));
}

Wait for Ajax call to finish and data to load

You can realise that sometimes it is not enough to wait for Ajax to finish rather than to wait for data to be rendered. There is fancy loader in my application under which is a DIV shown when some action is being performed. If there is one on your application then you’d better wait not only for Ajax to finish, but to loader to hide.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
private static void WaitForReady()
{
    WebDriverWait wait = new WebDriverWait(webDriver, waitForElement);
    wait.Until(driver =>
    {
        bool isAjaxFinished = (bool)((IJavaScriptExecutor)driver).
            ExecuteScript("return jQuery.active == 0");
        try
        {
            driver.FindElement(By.ClassName("spinner"));
            return false;
        }
        catch
        {
            return isAjaxFinished;
        }
    });
}

If “spinner” location gives exception then loader is not present and we can stop waiting. Good!

Improve the wait for data load

What about performance? When putting a timer the result was ~300ms for each Selenium search for loader. Not so good… Is 300ms long? Sure not, but taking into consideration this is called every time an element is located then this could make a huge difference in test execution times.

Why not making the same check for hidden loader, but this time with a JavaScript call to browser? I’m familiar with jQuery, then why not.

1
2
3
4
5
6
7
8
9
10
11
12
private static void WaitForReady()
{
    WebDriverWait wait = new WebDriverWait(webDriver, waitForElement);
    wait.Until(driver =>
    {
        bool isAjaxFinished = (bool)((IJavaScriptExecutor)driver).
            ExecuteScript("return jQuery.active == 0");
        bool isLoaderHidden = (bool)((IJavaScriptExecutor)driver).
            ExecuteScript("return $('.spinner').is(':visible') == false");
        return isAjaxFinished & isLoaderHidden;
    });
}

Conclusion

Same logic to check that element with class=”spinner” is not visible on page but this time at a cost of ~30ms. I like it much better this way!

Export a sqlite data to CSV

import pandas as pd

import sqlite3

conn = sqlite3.connect(‘test.db’)

cursor = conn.cursor()
cursor.execute(“SELECT name FROM sqlite_master WHERE type=’table’;”)
tables = cursor.fetchall()
for table_name in table_list:
try:
# table_name = table_name[0]
table = pd.read_sql_query(“SELECT * from %s” % table_name, conn)
table.to_csv(os.path.join(data_path, ‘short_karen2’, ‘{}.csv’.format(table_name)), index_label=’index’)
except Exception:
pass