Your ROOT_URL in app.ini is http://192.168.1.7:3000/ but you are visiting http://shqkxh.org:3000/guofu/dolphin-dev/blame/commit/083c66abb1c7e165b2296828b1b77f0e6a8fbb41/ipynb/ddb.ipynb You should set ROOT_URL correctly, otherwise the web may not work correctly.
guofu
/
dolphin-dev
1
0
Fork 0
Code Issues 3 Pull Requests Packages Projects Releases 1 Wiki Activity
dolphin-dev/ipynb/ddb.ipynb

2384 lines
80 KiB
Raw Normal View History Unescape

1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
3 years ago
{
"cells": [
{
"cell_type": "code",
Dump daily one-hot concept data to dolphindb.
3 years ago
"execution_count": 3,
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
3 years ago
"id": "7246e0c8-61cd-4cbf-a978-aa0dc0172d6d",
"metadata": {},
"outputs": [],
"source": [
"import dolphindb as ddb"
]
},
{
"cell_type": "code",
1. Reorganized the directory structure. 2. Add `DDBBase` class to decrease repeated class-level constant definitino. 3. Add `DDBExpression` class for single quarter calculation and yoy claculation.
3 years ago
"execution_count": 3,
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
3 years ago
"id": "5d0f471e-682e-43cc-abdb-7e52f3bbd707",
"metadata": {},
"outputs": [],
"source": [
1. Reorganized the directory structure. 2. Add `DDBBase` class to decrease repeated class-level constant definitino. 3. Add `DDBExpression` class for single quarter calculation and yoy claculation.
3 years ago
"sess = ddb.session('localhost', 8848)\n",
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
3 years ago
"sess.login('admin', '123456')"
]
},
1. Reorganized the directory structure. 2. Add `DDBBase` class to decrease repeated class-level constant definitino. 3. Add `DDBExpression` class for single quarter calculation and yoy claculation.
3 years ago
{
"cell_type": "code",
"execution_count": null,
"id": "fca58bdc-2aa1-4610-9a94-67d55f97a6e1",
"metadata": {},
"outputs": [],
"source": [
"sess.run(\"\"\"\n",
" migrate('/data/dolphindb/backup/', \"dfs://hft_stock_ts\", \"OrderPartitioned\")\n",
"\"\"\")"
]
},
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
3 years ago
{
"cell_type": "code",
Dump daily one-hot concept data to dolphindb.
3 years ago
"execution_count": null,
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
3 years ago
"id": "ed17fd0b-9b36-47e4-9ab6-11459a3621fb",
"metadata": {},
Dump daily one-hot concept data to dolphindb.
3 years ago
"outputs": [],
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
3 years ago
"source": [
Update README.md
3 years ago
"sess = ddb.session('192.168.1.7', 8848)\n",
"sess.login('admin', '123456')\n",
"\n",
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
3 years ago
"# backup(backup_path, sql_obj, force, parallel)\n",
"code = \"\"\"\n",
Dump daily one-hot concept data to dolphindb.
3 years ago
" backup('/data/dolphindb/backup/', <select * from loadTable(\"dfs://hft_stock_ts\", \"TransePartitioned\")>, false, false)\n",
Update README.md
3 years ago
"\"\"\"\n",
"\n",
"sess.run(code)"
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
3 years ago
]
},
{
"cell_type": "code",
Dump daily one-hot concept data to dolphindb.
3 years ago
"execution_count": 127,
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
"id": "c8d07fc8-d80c-490f-9220-0d3e8e4c72a4",
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
3 years ago
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
"50"
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
3 years ago
]
},
Dump daily one-hot concept data to dolphindb.
3 years ago
"execution_count": 127,
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
3 years ago
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
"sess = ddb.session('localhost', 8848)\n",
"sess.login('admin', '123456')\n",
"\n",
"# backup(backup_path, sql_obj, force, parallel)\n",
"code = \"\"\"\n",
" backup('/data/dolphindb/backup/', <select * from loadTable(\"dfs://pit_stock_ts\", \"bs_common_ori\")>, true, false);\n",
" backup('/data/dolphindb/backup/', <select * from loadTable(\"dfs://pit_stock_ts\", \"bs_common_adj\")>, true, false);\n",
" \n",
" backup('/data/dolphindb/backup/', <select * from loadTable(\"dfs://pit_stock_ts\", \"cfs_common_ori\")>, true, false);\n",
" backup('/data/dolphindb/backup/', <select * from loadTable(\"dfs://pit_stock_ts\", \"cfs_common_adj\")>, true, false);\n",
" \n",
" backup('/data/dolphindb/backup/', <select * from loadTable(\"dfs://pit_stock_ts\", \"is_common_adj\")>, true, false);\n",
" backup('/data/dolphindb/backup/', <select * from loadTable(\"dfs://pit_stock_ts\", \"is_common_adj\")>, true, false);\n",
" \n",
" backup('/data/dolphindb/backup/', <select * from loadTable(\"dfs://pit_stock_ts\", \"divident\")>, true, false);\n",
" backup('/data/dolphindb/backup/', <select * from loadTable(\"dfs://pit_stock_ts\", \"earnings_preannouncement\")>, true, false);\n",
" backup('/data/dolphindb/backup/', <select * from loadTable(\"dfs://pit_stock_ts\", \"preliminary_earnings_estimate\")>, true, false);\n",
"\"\"\"\n",
"\n",
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
3 years ago
"sess.run(code)"
]
},
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
{
"cell_type": "code",
Dump daily one-hot concept data to dolphindb.
3 years ago
"execution_count": 2,
"id": "45c4cbc3-08a1-4083-8117-44e8e3364375",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Failed to connect to host = localhost port = 8848 with error code 111\n"
]
},
{
"ename": "RuntimeError",
"evalue": "<Exception> in login: Couldn't send script/function to the remote host because the connection has been closed",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mRuntimeError\u001b[0m Traceback (most recent call last)",
"Input \u001b[0;32mIn [2]\u001b[0m, in \u001b[0;36m<cell line: 3>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mdolphindb\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m \u001b[38;5;21;01mddb\u001b[39;00m\n\u001b[1;32m 2\u001b[0m sess \u001b[38;5;241m=\u001b[39m ddb\u001b[38;5;241m.\u001b[39msession(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mlocalhost\u001b[39m\u001b[38;5;124m'\u001b[39m, \u001b[38;5;241m8848\u001b[39m)\n\u001b[0;32m----> 3\u001b[0m \u001b[43msess\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mlogin\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43madmin\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43m123456\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[1;32m 5\u001b[0m \u001b[38;5;66;03m# backup(backup_path, sql_obj, force, parallel)\u001b[39;00m\n\u001b[1;32m 6\u001b[0m sess\u001b[38;5;241m.\u001b[39mrun(\u001b[38;5;124m\"\"\"\u001b[39m\n\u001b[1;32m 7\u001b[0m \u001b[38;5;124m backup(\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m/data/dolphindb/backup/\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m, <select * from loadTable(\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mdfs://daily_stock_ts\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m, \u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mhft_daily_factor\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m)>, false, false);\u001b[39m\n\u001b[1;32m 8\u001b[0m \u001b[38;5;124m backup(\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m/data/dolphindb/backup/\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m, <select * from loadTable(\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mdfs://daily_stock_ts\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m, \u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124midx_daily_concept\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m)>, false, false);\u001b[39m\n\u001b[1;32m 9\u001b[0m \u001b[38;5;124m backup(\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m/data/dolphindb/backup/\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m, <select * from loadTable(\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mdfs://daily_stock_ts\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m, \u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mdaily_kline\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m)>, false, false);\u001b[39m\n\u001b[1;32m 10\u001b[0m \u001b[38;5;124m\"\"\"\u001b[39m)\n",
"File \u001b[0;32m~/.venv/tinysoft/lib/python3.8/site-packages/dolphindb/session.py:142\u001b[0m, in \u001b[0;36msession.login\u001b[0;34m(self, userid, password, enableEncryption)\u001b[0m\n\u001b[1;32m 140\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mpassword \u001b[38;5;241m=\u001b[39m password\n\u001b[1;32m 141\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39menableEncryption \u001b[38;5;241m=\u001b[39m enableEncryption\n\u001b[0;32m--> 142\u001b[0m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcpp\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mlogin\u001b[49m\u001b[43m(\u001b[49m\u001b[43muserid\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mpassword\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43menableEncryption\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 143\u001b[0m \u001b[38;5;28;01mfinally\u001b[39;00m:\n\u001b[1;32m 144\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mmutex\u001b[38;5;241m.\u001b[39mrelease()\n",
"\u001b[0;31mRuntimeError\u001b[0m: <Exception> in login: Couldn't send script/function to the remote host because the connection has been closed"
]
}
],
"source": [
"import dolphindb as ddb\n",
"sess = ddb.session('localhost', 8848)\n",
"sess.login('admin', '123456')\n",
"\n",
"# backup(backup_path, sql_obj, force, parallel)\n",
"sess.run(\"\"\"\n",
" backup('/data/dolphindb/backup/', <select * from loadTable(\"dfs://daily_stock_ts\", \"hft_daily_factor\")>, false, false);\n",
" backup('/data/dolphindb/backup/', <select * from loadTable(\"dfs://daily_stock_ts\", \"idx_daily_concept\")>, false, false);\n",
" backup('/data/dolphindb/backup/', <select * from loadTable(\"dfs://daily_stock_ts\", \"daily_kline\")>, false, false);\n",
"\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "463a59a8-39cf-4406-9390-da5dfdc44fa7",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"50"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sess = ddb.session('192.168.1.7', 8848)\n",
"sess.login('admin', '123456')\n",
"\n",
"sess.run(\"\"\"\n",
" backup('/data/dolphindb/backup/', <select * from loadTable(\"dfs://daily_stock_ts\", \"idx_daily_concept\")>, false, false);\n",
"\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "84f99b12-e868-425e-bcd3-1733feea7524",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>dbName</th>\n",
" <th>tableName</th>\n",
" <th>success</th>\n",
" <th>errorMsg</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>dfs://daily_stock_ts</td>\n",
" <td>idx_daily_concept</td>\n",
" <td>True</td>\n",
" <td></td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" dbName tableName success errorMsg\n",
"0 dfs://daily_stock_ts idx_daily_concept True "
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sess = ddb.session('localhost', 8848)\n",
"sess.login('admin', '123456')\n",
"\n",
"sess.run(\"\"\"\n",
" dropTable(database(\"dfs://daily_stock_ts\"), \"idx_daily_concept\");\n",
" migrate('/data/dolphindb/backup/', \"dfs://daily_stock_ts\", \"idx_daily_concept\");\n",
"\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": 13,
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
"id": "8b7dae3d-aef1-4c50-92b2-460d4fea0a96",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>tableName</th>\n",
" <th>physicalIndex</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
Dump daily one-hot concept data to dolphindb.
3 years ago
" <td>idx_daily_concept</td>\n",
" <td>1esb</td>\n",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>hft_daily_factor</td>\n",
" <td>u6J</td>\n",
" </tr>\n",
Dump daily one-hot concept data to dolphindb.
3 years ago
" <tr>\n",
" <th>2</th>\n",
" <td>daily_kline</td>\n",
" <td>uoH</td>\n",
" </tr>\n",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
Dump daily one-hot concept data to dolphindb.
3 years ago
" tableName physicalIndex\n",
"0 idx_daily_concept 1esb\n",
"1 hft_daily_factor u6J\n",
"2 daily_kline uoH"
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
]
},
Dump daily one-hot concept data to dolphindb.
3 years ago
"execution_count": 13,
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
Dump daily one-hot concept data to dolphindb.
3 years ago
"sess = ddb.session('localhost', 8848)\n",
"sess.login('admin', '123456')\n",
"\n",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
"sess.run(\"\"\"\n",
Dump daily one-hot concept data to dolphindb.
3 years ago
" //getAllDBs()\n",
" //listTables(\"dfs://info_stock_ts\");\n",
" //listTables(\"dfs://hft_stock_ts\");\n",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
" listTables(\"dfs://daily_stock_ts\");\n",
Dump daily one-hot concept data to dolphindb.
3 years ago
" //schema(loadTable(\"dfs://daily_stock_ts\", \"idx_daily_concept\"))\n",
"\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "fb23b78d-c402-4c11-9504-c61793e5a2a3",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'partitionType': 5,\n",
" 'partitionColumnType': 17,\n",
" 'partitionColumnIndex': 0,\n",
" 'chunkPath': None,\n",
" 'colDefs': name typeString typeInt comment\n",
" 0 code SYMBOL 17 \n",
" 1 m_nDate DATE 6 \n",
" 2 open DOUBLE 16 \n",
" 3 high DOUBLE 16 \n",
" 4 low DOUBLE 16 \n",
" 5 close DOUBLE 16 \n",
" 6 vol DOUBLE 16 \n",
" 7 amount DOUBLE 16 \n",
" 8 cjbs INT 4 \n",
" 9 yclose DOUBLE 16 \n",
" 10 PctChg DOUBLE 16 \n",
" 11 IsZt INT 4 \n",
" 12 IsDt INT 4 \n",
" 13 IsST INT 4 \n",
" 14 IsGoDelist INT 4 \n",
" 15 FloatShares DOUBLE 16 \n",
" 16 MarketValues DOUBLE 16 \n",
" 17 factor DOUBLE 16 ,\n",
" 'chunkGranularity': 'TABLE',\n",
" 'partitionTypeName': 'HASH',\n",
" 'keepDuplicates': 'ALL',\n",
" 'engineType': 'TSDB',\n",
" 'partitionColumnName': 'code',\n",
" 'partitionSchema': 50,\n",
" 'sortColumns': array(['code', 'm_nDate'], dtype=object),\n",
" 'partitionSites': None}"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sess.run(\"\"\"\n",
" schema(loadTable(\"dfs://daily_stock_ts\", \"daily_kline\"))\n",
"\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": 84,
"id": "84e9e3df-aec7-4edf-96fc-2481ed6790b5",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>dbName</th>\n",
" <th>tableName</th>\n",
" <th>success</th>\n",
" <th>errorMsg</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>dfs://daily_stock_ts</td>\n",
" <td>idx_daily_concept</td>\n",
" <td>True</td>\n",
" <td></td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" dbName tableName success errorMsg\n",
"0 dfs://daily_stock_ts idx_daily_concept True "
]
},
"execution_count": 84,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sess = ddb.session('192.168.1.7', 8848)\n",
"sess.login('admin', '123456')\n",
"sess.run(\"\"\"\n",
" migrate('/data/dolphindb/backup/', \"dfs://daily_stock_ts\", \"idx_daily_concept\")\n",
"\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "58ac1e6f-51cd-4d35-a34f-8220fca52acb",
"metadata": {},
"outputs": [],
"source": [
"sess.run(\"\"\"\n",
" \n",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
"\"\"\")"
]
},
Dump daily one-hot concept data to dolphindb.
3 years ago
{
"cell_type": "code",
"execution_count": 16,
"id": "3411a78a-4efc-4693-af3f-44abe6ba166e",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>min_m_nDate</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>2006-01-04</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" min_m_nDate\n",
"0 2006-01-04"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sess.run(\"select min(m_nDate) from loadTable('dfs://daily_stock_ts', 'daily_kline')\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4a5ddd8f-0bb2-4427-b5af-b32d80d6b3cf",
"metadata": {},
"outputs": [],
"source": [
"sess.run(\"\"\"\n",
" tbl = loadTable(\"dfs://hft_stock_ts\", \"KLinePartitioned\");\n",
" select * from (\n",
" select code, m_nDate, count(*) from tbl where code='600009.SH' group by code, m_nDate map\n",
" );\n",
"\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": 131,
"id": "06efcf17-b210-452f-a5fb-d4f97b5eae6b",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"10"
]
},
"execution_count": 131,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sess = ddb.session('localhost', 8848)\n",
"sess.login('admin', '123456')\n",
"\n",
"# backup(backup_path, sql_obj, force, parallel)\n",
"code = \"\"\"\n",
" backup('/data/dolphindb/backup/', <select * from loadTable(\"dfs://info_stock_ts\", \"info_list_delist\")>, true, false);\n",
" backup('/data/dolphindb/backup/', <select * from loadTable(\"dfs://info_stock_ts\", \"info_industry\")>, true, false);\n",
"\"\"\"\n",
"\n",
"sess.run(code)"
]
},
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
{
"cell_type": "code",
"execution_count": 34,
"id": "8af3381d-9fdc-4cb0-b3a9-d6538db97476",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"50"
]
},
"execution_count": 34,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sess.run(\"\"\"\n",
" backup('/data/dolphindb/backup/', <select * from loadTable(\"dfs://daily_stock_ts\", \"daily_kline\") >, true, false);\n",
"\"\"\")"
]
},
1. `DDBDailyFactor.py` => `DDBFactor.py` 2. 在`DDBLoader.py`中增加了`DDBPITLoader`类，专门用于向DolphinDB里面添加PIT数据 3. 其他一些优化
3 years ago
{
"cell_type": "code",
"execution_count": 69,
"id": "bf3bc38e-74cb-4549-bdca-6a3d2a601488",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"2287.0"
]
},
"execution_count": 69,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"114350 / 50"
]
},
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
3 years ago
{
"cell_type": "code",
"execution_count": 6,
"id": "11275975-af43-4123-973a-f75096eb4e43",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"2311.0"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"115550 / 50"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "155735c7-bdad-4c64-a2a1-514fe54e088c",
"metadata": {},
"outputs": [],
"source": [
"import dolphindb as ddb\n",
"\n",
"sess = ddb.session('localhost', 8848)\n",
"sess.login('admin', '123456')"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "a2e95bc5-c1d5-47f5-9488-9b8702947afc",
"metadata": {},
"outputs": [],
"source": [
"# migrate(backupDir, [backupDBPath], [backupTableName], [newDBPath=backupDBPath], [newTableName=backupTableName])\n",
"code = \"\"\"\n",
" migrate('/data/dolphindb/backup/',\"dfs://hft_stock_ts\", \"KLinePartitioned\")\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8a6198ef-e7f1-4f58-ba24-32dc429c9803",
"metadata": {},
"outputs": [],
"source": [
"sess.run(code)"
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "60f71516-33fc-4147-9740-c493ccb6b6a8",
"metadata": {},
"outputs": [],
"source": [
"def make_kurto_code_partition(partition_num):\n",
" code = \"\"\"\n",
" tbl = loadTable(\"dfs://hft_stock_ts\", \"KLinePartitioned\");\n",
" ret = select \n",
" code, m_nDate, eachPre(\\, m_nClose)-1.0 as ret \n",
" from tbl \n",
" where partition(code, {partition_num}) \n",
" context by m_nDate;\n",
" kurto = select \n",
" code, m_nDate, sqrt(239) * sum(pow(ret, 3)) / pow(sum(pow(ret, 2)), 1.5) as kurto \n",
" from ret \n",
" group by code, m_nDate;\n",
" \"\"\".format(\n",
" partition_num = partition_num\n",
" )\n",
" sess.run(code)"
]
},
{
"cell_type": "code",
"execution_count": 32,
"id": "1397c1b7-f81a-465e-868a-894b3ef818fc",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>code</th>\n",
" <th>count</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>000400.SZ</td>\n",
" <td>2305</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>000558.SZ</td>\n",
" <td>2063</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>000602.SZ</td>\n",
" <td>47</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>000677.SZ</td>\n",
" <td>2009</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>000731.SZ</td>\n",
" <td>2311</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>94</th>\n",
" <td>688039.SH</td>\n",
" <td>624</td>\n",
" </tr>\n",
" <tr>\n",
" <th>95</th>\n",
" <td>688052.SH</td>\n",
" <td>52</td>\n",
" </tr>\n",
" <tr>\n",
" <th>96</th>\n",
" <td>688091.SH</td>\n",
" <td>198</td>\n",
" </tr>\n",
" <tr>\n",
" <th>97</th>\n",
" <td>688368.SH</td>\n",
" <td>656</td>\n",
" </tr>\n",
" <tr>\n",
" <th>98</th>\n",
" <td>688396.SH</td>\n",
" <td>575</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>99 rows × 2 columns</p>\n",
"</div>"
],
"text/plain": [
" code count\n",
"0 000400.SZ 2305\n",
"1 000558.SZ 2063\n",
"2 000602.SZ 47\n",
"3 000677.SZ 2009\n",
"4 000731.SZ 2311\n",
".. ... ...\n",
"94 688039.SH 624\n",
"95 688052.SH 52\n",
"96 688091.SH 198\n",
"97 688368.SH 656\n",
"98 688396.SH 575\n",
"\n",
"[99 rows x 2 columns]"
]
},
"execution_count": 32,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sess.run(\"\"\"\n",
" select code, count(*) from kurto group by code\n",
"\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "59758901-3728-4355-bbb8-15b3eb537cdf",
"metadata": {},
"outputs": [],
"source": [
"def create_daily_ddb_database():\n",
" code = \"\"\"\n",
" daily_stock_ts = database(\n",
" \"dfs://daily_stock_ts\", \n",
" HASH, [SYMBOL, 50], \n",
" engine = 'TSDB'\n",
" )\n",
" \"\"\"\n",
" sess.run(code)"
]
},
{
"cell_type": "code",
"execution_count": 38,
"id": "d8fa94ae-6ca4-41f3-8b13-47ab8876ba95",
"metadata": {},
"outputs": [],
"source": [
"def load_daily_ddb_database():\n",
" code = \"\"\"\n",
" daily_stock_ts = database(\"dfs://daily_stock_ts\")\n",
" \"\"\"\n",
" sess.run(code)\n",
"\n",
"load_daily_ddb_database()"
]
},
{
"cell_type": "code",
"execution_count": 43,
"id": "84eb237e-959c-419f-bce1-6779acda86d4",
"metadata": {},
"outputs": [],
"source": [
"def create_kurto_partition_table():\n",
" # createPartitionedTable(dbHandle, table, tableName, [partitionColumns], [compressMethods], [sortColumns], [keepDuplicates=ALL], [sortKeyMappingFunction])\n",
" code = \"\"\"\n",
" dropTable(daily_stock_ts, \"kurto_partitioned\");\n",
" kurto_partitioned = createPartitionedTable(\n",
" daily_stock_ts,\n",
" kurto, \n",
" \"kurto_partitioned\",\n",
" partitionColumns = 'code',\n",
" compressMethods = {'m_nDate' : 'delta'},\n",
" sortColumns = `code`m_nDate\n",
" );\n",
" \"\"\"\n",
" sess.run(code)\n",
" \n",
"create_kurto_partition_table()"
]
},
{
"cell_type": "code",
"execution_count": 40,
"id": "0d93b3d1-39b7-4e23-8abf-eda86dcd2a23",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>code</th>\n",
" <th>m_nDate</th>\n",
" <th>kurto</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>000400.SZ</td>\n",
" <td>2013-01-04</td>\n",
" <td>-0.317991</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>000558.SZ</td>\n",
" <td>2013-01-04</td>\n",
" <td>-15.387820</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>000677.SZ</td>\n",
" <td>2013-01-04</td>\n",
" <td>-15.459625</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>000731.SZ</td>\n",
" <td>2013-01-04</td>\n",
" <td>15.443339</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>000752.SZ</td>\n",
" <td>2013-01-04</td>\n",
" <td>15.396262</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>000789.SZ</td>\n",
" <td>2013-01-04</td>\n",
" <td>-14.801720</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>000903.SZ</td>\n",
" <td>2013-01-04</td>\n",
" <td>-15.405085</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>000912.SZ</td>\n",
" <td>2013-01-04</td>\n",
" <td>14.755241</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>000935.SZ</td>\n",
" <td>2013-01-04</td>\n",
" <td>15.413214</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>000970.SZ</td>\n",
" <td>2013-01-04</td>\n",
" <td>15.458413</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" code m_nDate kurto\n",
"0 000400.SZ 2013-01-04 -0.317991\n",
"1 000558.SZ 2013-01-04 -15.387820\n",
"2 000677.SZ 2013-01-04 -15.459625\n",
"3 000731.SZ 2013-01-04 15.443339\n",
"4 000752.SZ 2013-01-04 15.396262\n",
"5 000789.SZ 2013-01-04 -14.801720\n",
"6 000903.SZ 2013-01-04 -15.405085\n",
"7 000912.SZ 2013-01-04 14.755241\n",
"8 000935.SZ 2013-01-04 15.413214\n",
"9 000970.SZ 2013-01-04 15.458413"
]
},
"execution_count": 40,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sess.run(\"\"\"\n",
" kurto_partitioned.tableInsert(kurto);\n",
"\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": 41,
"id": "eea5cb0b-8452-41a9-a0ad-7cf52c856c25",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>count</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>154953</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" count\n",
"0 154953"
]
},
"execution_count": 41,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sess.run(\"\"\"\n",
" select count(*) from kurto_partitioned;\n",
"\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": 45,
"id": "9e7bc856-517b-4bf9-866d-4fe85b9705a3",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'partitionType': 5,\n",
" 'partitionColumnType': 17,\n",
" 'partitionColumnIndex': 0,\n",
" 'chunkPath': None,\n",
" 'colDefs': name typeString typeInt comment\n",
" 0 code SYMBOL 17 \n",
" 1 m_nDate DATE 6 \n",
" 2 kurto DOUBLE 16 ,\n",
" 'chunkGranularity': 'TABLE',\n",
" 'partitionTypeName': 'HASH',\n",
" 'keepDuplicates': 'ALL',\n",
" 'engineType': 'TSDB',\n",
" 'partitionColumnName': 'code',\n",
" 'partitionSchema': 50,\n",
" 'sortColumns': array(['code', 'm_nDate'], dtype=object),\n",
" 'partitionSites': None}"
]
},
"execution_count": 45,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sess.run(\"\"\"\n",
" schema(loadTable(\"dfs://daily_stock_ts\", \"kurto_partitioned\"));\n",
"\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": 46,
"id": "94ae00b5-ba42-48b9-b114-012cf306d871",
"metadata": {},
"outputs": [],
"source": [
"sess.run(\"m_table = table(kurto)\")"
]
},
{
"cell_type": "code",
"execution_count": 47,
"id": "e30134f9-5110-421d-8f71-1bef25e7ef17",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>code</th>\n",
" <th>m_nDate</th>\n",
" <th>kurto</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>000400.SZ</td>\n",
" <td>2013-01-04</td>\n",
" <td>-0.317991</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>000400.SZ</td>\n",
" <td>2013-01-07</td>\n",
" <td>0.790872</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>000400.SZ</td>\n",
" <td>2013-01-08</td>\n",
" <td>0.655005</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>000400.SZ</td>\n",
" <td>2013-01-09</td>\n",
" <td>0.965054</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>000400.SZ</td>\n",
" <td>2013-01-10</td>\n",
" <td>-0.116488</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>154948</th>\n",
" <td>688396.SH</td>\n",
" <td>2022-07-04</td>\n",
" <td>-15.420192</td>\n",
" </tr>\n",
" <tr>\n",
" <th>154949</th>\n",
" <td>688396.SH</td>\n",
" <td>2022-07-05</td>\n",
" <td>-15.426922</td>\n",
" </tr>\n",
" <tr>\n",
" <th>154950</th>\n",
" <td>688396.SH</td>\n",
" <td>2022-07-06</td>\n",
" <td>-15.389434</td>\n",
" </tr>\n",
" <tr>\n",
" <th>154951</th>\n",
" <td>688396.SH</td>\n",
" <td>2022-07-07</td>\n",
" <td>-15.411602</td>\n",
" </tr>\n",
" <tr>\n",
" <th>154952</th>\n",
" <td>688396.SH</td>\n",
" <td>2022-07-08</td>\n",
" <td>-15.401453</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>154953 rows × 3 columns</p>\n",
"</div>"
],
"text/plain": [
" code m_nDate kurto\n",
"0 000400.SZ 2013-01-04 -0.317991\n",
"1 000400.SZ 2013-01-07 0.790872\n",
"2 000400.SZ 2013-01-08 0.655005\n",
"3 000400.SZ 2013-01-09 0.965054\n",
"4 000400.SZ 2013-01-10 -0.116488\n",
"... ... ... ...\n",
"154948 688396.SH 2022-07-04 -15.420192\n",
"154949 688396.SH 2022-07-05 -15.426922\n",
"154950 688396.SH 2022-07-06 -15.389434\n",
"154951 688396.SH 2022-07-07 -15.411602\n",
"154952 688396.SH 2022-07-08 -15.401453\n",
"\n",
"[154953 rows x 3 columns]"
]
},
"execution_count": 47,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sess.run(\"m_table\")"
]
},
{
"cell_type": "code",
"execution_count": 51,
"id": "947e3346-15f6-4fb5-b2d6-b97b013a3b36",
"metadata": {},
"outputs": [],
"source": [
"code = \"\"\"\n",
" // 保证创建新的分区表不会和已经存在表冲突\n",
" if (existsTable(\"{ddb_daily_path}\", \"{partition_tbl_name}\")) {{\n",
" dropTable({ddb_daily_dbname}, \"{partition_tbl_name}\");\n",
" }}\n",
"\n",
" {partition_tbl_name} = createPartitionedTable(\n",
" dbHandle = {ddb_daily_dbname},\n",
" table = {memory_tbl_name}, \n",
" \"{partition_tbl_name}\",\n",
" partitionColumns = 'code',\n",
" compressMethods = {{'m_nDate' : 'delta'}},\n",
" sortColumns = `code`m_nDate\n",
" );\n",
"\"\"\".format(\n",
" ddb_daily_path = \"A\",\n",
" ddb_daily_dbname = \"B\",\n",
" partition_tbl_name = \"C\",\n",
" memory_tbl_name = \"D\",\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 53,
"id": "5d758c68-301a-47a7-9cd2-b7dd7edf1a55",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"('\\n'\n",
" ' // 保证创建新的分区表不会和已经存在表冲突\\n'\n",
" ' if (existsTable(\"A\", \"C\")) {\\n'\n",
" ' dropTable(B, \"C\");\\n'\n",
" ' }\\n'\n",
" '\\n'\n",
" ' C = createPartitionedTable(\\n'\n",
" ' dbHandle = B,\\n'\n",
" ' table = D, \\n'\n",
" ' \"C\",\\n'\n",
" \" partitionColumns = 'code',\\n\"\n",
" \" compressMethods = {'m_nDate' : 'delta'},\\n\"\n",
" ' sortColumns = `code`m_nDate\\n'\n",
" ' );\\n')\n"
]
}
],
"source": [
"from pprint import pprint\n",
"pprint(code)"
]
},
{
"cell_type": "code",
"execution_count": 56,
"id": "400424d9-9c58-4417-94e4-fe1569522b03",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'chunkGranularity': 'TABLE',\n",
" 'chunkPath': None,\n",
" 'colDefs': name typeString typeInt comment\n",
"0 code SYMBOL 17 \n",
"1 m_nDate DATE 6 \n",
"2 kurto DOUBLE 16 ,\n",
" 'engineType': 'TSDB',\n",
" 'keepDuplicates': 'ALL',\n",
" 'partitionColumnIndex': 0,\n",
" 'partitionColumnName': 'code',\n",
" 'partitionColumnType': 17,\n",
" 'partitionSchema': 50,\n",
" 'partitionSites': None,\n",
" 'partitionType': 5,\n",
" 'partitionTypeName': 'HASH',\n",
" 'sortColumns': array(['code', 'm_nDate'], dtype=object)}\n"
]
}
],
"source": [
"pprint(\n",
" sess.run(\"\"\"\n",
" schema(loadTable(\"dfs://daily_stock_ts\", \"hft_daily_factor\"));\n",
" \"\"\")\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 62,
"id": "68e74de8-359c-4c47-a361-2c006896211c",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>count</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>309906</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" count\n",
"0 309906"
]
},
"execution_count": 62,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sess.run(\"\"\"\n",
" select count(*) from loadTable(\"dfs://daily_stock_ts\", \"hft_daily_factor\");\n",
"\"\"\")"
]
},
{
"cell_type": "code",
reorganize the directory structure
3 years ago
"execution_count": 21,
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
3 years ago
"id": "b1bd7605-c347-44f2-a90f-bc949cde0ef0",
"metadata": {},
reorganize the directory structure
3 years ago
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>code</th>\n",
" <th>report_period</th>\n",
" <th>appear_in_period</th>\n",
" <th>appear_at_date</th>\n",
" <th>TOT_OPER_REV</th>\n",
" <th>OPER_REV</th>\n",
" <th>INT_INC</th>\n",
" <th>INSUR_PREM_UNEARNED</th>\n",
" <th>HANDLING_CHRG_COMM_INC</th>\n",
" <th>NET_INC_OTHER_OPS</th>\n",
" <th>...</th>\n",
" <th>OTHER_INCOME</th>\n",
" <th>MEMO</th>\n",
" <th>ASSET_DISPOSAL_INCOME</th>\n",
" <th>CONTINUED_NET_PROFIT</th>\n",
" <th>END_NET_PROFIT</th>\n",
" <th>CREDIT_IMPAIRMENT_LOSS</th>\n",
" <th>RD_EXPENSE</th>\n",
" <th>STMNOTE_FINEXP</th>\n",
" <th>FIN_EXP_INT_INC</th>\n",
" <th>TOT_OPER_COST2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>000677.SZ</td>\n",
" <td>2009-12-31</td>\n",
" <td>2009-12-31</td>\n",
" <td>2010-04-26</td>\n",
" <td>3.514495e+09</td>\n",
" <td>3.514495e+09</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>26392229.68</td>\n",
" <td>...</td>\n",
" <td>0.0</td>\n",
" <td>nan</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>3.474455e+09</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>000677.SZ</td>\n",
" <td>2009-12-31</td>\n",
" <td>2009-12-31</td>\n",
" <td>2011-12-10</td>\n",
" <td>3.514495e+09</td>\n",
" <td>3.514495e+09</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>26392229.68</td>\n",
" <td>...</td>\n",
" <td>0.0</td>\n",
" <td>nan</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>3.460335e+09</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>2 rows × 69 columns</p>\n",
"</div>"
],
"text/plain": [
" code report_period appear_in_period appear_at_date TOT_OPER_REV \\\n",
"0 000677.SZ 2009-12-31 2009-12-31 2010-04-26 3.514495e+09 \n",
"1 000677.SZ 2009-12-31 2009-12-31 2011-12-10 3.514495e+09 \n",
"\n",
" OPER_REV INT_INC INSUR_PREM_UNEARNED HANDLING_CHRG_COMM_INC \\\n",
"0 3.514495e+09 0.0 0.0 0.0 \n",
"1 3.514495e+09 0.0 0.0 0.0 \n",
"\n",
" NET_INC_OTHER_OPS ... OTHER_INCOME MEMO ASSET_DISPOSAL_INCOME \\\n",
"0 26392229.68 ... 0.0 nan 0.0 \n",
"1 26392229.68 ... 0.0 nan 0.0 \n",
"\n",
" CONTINUED_NET_PROFIT END_NET_PROFIT CREDIT_IMPAIRMENT_LOSS RD_EXPENSE \\\n",
"0 0.0 0.0 0.0 0.0 \n",
"1 0.0 0.0 0.0 0.0 \n",
"\n",
" STMNOTE_FINEXP FIN_EXP_INT_INC TOT_OPER_COST2 \n",
"0 0.0 0.0 3.474455e+09 \n",
"1 0.0 0.0 3.460335e+09 \n",
"\n",
"[2 rows x 69 columns]"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import dolphindb as ddb \n",
"\n",
"sess = ddb.session('192.168.1.167', 8848)\n",
"sess.login('admin', '123456')\n",
"\n",
"sess.run(\"\"\"\n",
" is_common_adj = loadTable(\"dfs://pit_stock_ts\", \"is_common_adj\")\n",
" is_common_ori = loadTable(\"dfs://pit_stock_ts\", \"is_common_ori\")\n",
" select * from is_common_ori where code='000677.SZ' and report_period=2009.12.31\n",
"\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "f0b3928d-c06f-4976-b568-091d98293a6e",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>code</th>\n",
" <th>report_period</th>\n",
" <th>appear_in_period</th>\n",
" <th>appear_at_date</th>\n",
" <th>TOT_OPER_REV</th>\n",
" <th>OPER_REV</th>\n",
" <th>INT_INC</th>\n",
" <th>INSUR_PREM_UNEARNED</th>\n",
" <th>HANDLING_CHRG_COMM_INC</th>\n",
" <th>NET_INC_OTHER_OPS</th>\n",
" <th>...</th>\n",
" <th>OTHER_INCOME</th>\n",
" <th>MEMO</th>\n",
" <th>ASSET_DISPOSAL_INCOME</th>\n",
" <th>CONTINUED_NET_PROFIT</th>\n",
" <th>END_NET_PROFIT</th>\n",
" <th>CREDIT_IMPAIRMENT_LOSS</th>\n",
" <th>RD_EXPENSE</th>\n",
" <th>STMNOTE_FINEXP</th>\n",
" <th>FIN_EXP_INT_INC</th>\n",
" <th>TOT_OPER_COST2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>000677.SZ</td>\n",
" <td>2009-12-31</td>\n",
" <td>2010-12-31</td>\n",
" <td>2011-04-29</td>\n",
" <td>3.514495e+09</td>\n",
" <td>3.514495e+09</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>26392229.68</td>\n",
" <td>...</td>\n",
" <td>0.0</td>\n",
" <td>nan</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>3.474455e+09</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>1 rows × 69 columns</p>\n",
"</div>"
],
"text/plain": [
" code report_period appear_in_period appear_at_date TOT_OPER_REV \\\n",
"0 000677.SZ 2009-12-31 2010-12-31 2011-04-29 3.514495e+09 \n",
"\n",
" OPER_REV INT_INC INSUR_PREM_UNEARNED HANDLING_CHRG_COMM_INC \\\n",
"0 3.514495e+09 0.0 0.0 0.0 \n",
"\n",
" NET_INC_OTHER_OPS ... OTHER_INCOME MEMO ASSET_DISPOSAL_INCOME \\\n",
"0 26392229.68 ... 0.0 nan 0.0 \n",
"\n",
" CONTINUED_NET_PROFIT END_NET_PROFIT CREDIT_IMPAIRMENT_LOSS RD_EXPENSE \\\n",
"0 0.0 0.0 0.0 0.0 \n",
"\n",
" STMNOTE_FINEXP FIN_EXP_INT_INC TOT_OPER_COST2 \n",
"0 0.0 0.0 3.474455e+09 \n",
"\n",
"[1 rows x 69 columns]"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sess.run(\"select * from is_common_adj where code='000677.SZ' and report_period=2009.12.31\")"
]
},
{
"cell_type": "code",
Dump daily one-hot concept data to dolphindb.
3 years ago
"execution_count": 138,
reorganize the directory structure
3 years ago
"id": "d68ea326-82c3-4a7c-97cf-c04dd8aee56b",
"metadata": {},
"outputs": [
{
Dump daily one-hot concept data to dolphindb.
3 years ago
"ename": "RuntimeError",
"evalue": "<Exception> in run: Server response: 'tbl = loadTable(\"dfs://hft_stock_ts\", \"OrderPartitioned\") => FileBlock[/hft_stock_ts/domain, e56bb91f-d4d8-48be-6c4c-df3c1eb96dc4, {}, v1, 9413, c1] returned from name node didn't contain any site' script: '\ntbl = loadTable(\"dfs://hft_stock_ts\", 'OrderPartitioned');\nselect sum(cnt) from (select count(*) as cnt from tbl map);\n'",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mRuntimeError\u001b[0m Traceback (most recent call last)",
"Input \u001b[0;32mIn [138]\u001b[0m, in \u001b[0;36m<cell line: 3>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m sess \u001b[38;5;241m=\u001b[39m ddb\u001b[38;5;241m.\u001b[39msession(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m192.168.1.7\u001b[39m\u001b[38;5;124m'\u001b[39m, \u001b[38;5;241m8848\u001b[39m)\n\u001b[1;32m 2\u001b[0m sess\u001b[38;5;241m.\u001b[39mlogin(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124madmin\u001b[39m\u001b[38;5;124m'\u001b[39m, \u001b[38;5;124m'\u001b[39m\u001b[38;5;124m123456\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[0;32m----> 3\u001b[0m \u001b[43msess\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrun\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m\"\"\"\u001b[39;49m\n\u001b[1;32m 4\u001b[0m \u001b[38;5;124;43mtbl = loadTable(\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mdfs://hft_stock_ts\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43m, \u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mOrderPartitioned\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43m);\u001b[39;49m\n\u001b[1;32m 5\u001b[0m \u001b[38;5;124;43mselect sum(cnt) from (select count(*) as cnt from tbl map);\u001b[39;49m\n\u001b[1;32m 6\u001b[0m \u001b[38;5;124;43m\"\"\"\u001b[39;49m\u001b[43m)\u001b[49m\n",
"File \u001b[0;32m~/.venv/tinysoft/lib/python3.8/site-packages/dolphindb/session.py:161\u001b[0m, in \u001b[0;36msession.run\u001b[0;34m(self, script, *args, **kwargs)\u001b[0m\n\u001b[1;32m 159\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mfetchSize\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;129;01min\u001b[39;00m kwargs\u001b[38;5;241m.\u001b[39mkeys():\n\u001b[1;32m 160\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m BlockReader(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcpp\u001b[38;5;241m.\u001b[39mrunBlock(script, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs))\n\u001b[0;32m--> 161\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcpp\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrun\u001b[49m\u001b[43m(\u001b[49m\u001b[43mscript\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
"\u001b[0;31mRuntimeError\u001b[0m: <Exception> in run: Server response: 'tbl = loadTable(\"dfs://hft_stock_ts\", \"OrderPartitioned\") => FileBlock[/hft_stock_ts/domain, e56bb91f-d4d8-48be-6c4c-df3c1eb96dc4, {}, v1, 9413, c1] returned from name node didn't contain any site' script: '\ntbl = loadTable(\"dfs://hft_stock_ts\", 'OrderPartitioned');\nselect sum(cnt) from (select count(*) as cnt from tbl map);\n'"
]
reorganize the directory structure
3 years ago
}
],
"source": [
"sess = ddb.session('192.168.1.7', 8848)\n",
"sess.login('admin', '123456')\n",
"sess.run(\"\"\"\n",
Dump daily one-hot concept data to dolphindb.
3 years ago
"tbl = loadTable(\"dfs://hft_stock_ts\", 'OrderPartitioned');\n",
1. Reorganized the directory structure. 2. Add `DDBBase` class to decrease repeated class-level constant definitino. 3. Add `DDBExpression` class for single quarter calculation and yoy claculation.
3 years ago
"select sum(cnt) from (select count(*) as cnt from tbl map);\n",
"\"\"\")"
]
},
{
"cell_type": "code",
Dump daily one-hot concept data to dolphindb.
3 years ago
"execution_count": 20,
1. Reorganized the directory structure. 2. Add `DDBBase` class to decrease repeated class-level constant definitino. 3. Add `DDBExpression` class for single quarter calculation and yoy claculation.
3 years ago
"id": "7cbc4906-7756-424a-9ce5-9d2b6d1bab4b",
"metadata": {},
"outputs": [
{
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>sum_cnt</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
Dump daily one-hot concept data to dolphindb.
3 years ago
" <td>1483910</td>\n",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
Dump daily one-hot concept data to dolphindb.
3 years ago
" sum_cnt\n",
"0 1483910"
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
]
},
Dump daily one-hot concept data to dolphindb.
3 years ago
"execution_count": 20,
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
"metadata": {},
"output_type": "execute_result"
1. Reorganized the directory structure. 2. Add `DDBBase` class to decrease repeated class-level constant definitino. 3. Add `DDBExpression` class for single quarter calculation and yoy claculation.
3 years ago
}
],
"source": [
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
"sess = ddb.session('192.168.1.7', 8848)\n",
1. Reorganized the directory structure. 2. Add `DDBBase` class to decrease repeated class-level constant definitino. 3. Add `DDBExpression` class for single quarter calculation and yoy claculation.
3 years ago
"sess.login('admin', '123456')\n",
"sess.run(\"\"\"\n",
Dump daily one-hot concept data to dolphindb.
3 years ago
" tbl = loadTable(\"dfs://hft_stock_ts\", 'TickQueuePartitioned');\n",
" select sum(cnt) from (select count(*) as cnt from tbl where code='689009.SH' map);\n",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
"\"\"\")"
]
},
{
"cell_type": "code",
Dump daily one-hot concept data to dolphindb.
3 years ago
"execution_count": 9,
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
"id": "e9ab5e57-dce5-4426-9bac-4238cd067197",
"metadata": {},
"outputs": [
Dump daily one-hot concept data to dolphindb.
3 years ago
{
"name": "stdout",
"output_type": "stream",
"text": [
"Will retrieve partition 0\n",
"Will retrieve partition 1\n",
"Will retrieve partition 2\n",
"Will retrieve partition 3\n",
"Will retrieve partition 4\n",
"Will retrieve partition 5\n",
"Will retrieve partition 6\n",
"Will retrieve partition 7\n",
"Will retrieve partition 8\n",
"Will retrieve partition 9\n",
"Will retrieve partition 10\n",
"Will retrieve partition 11\n",
"Will retrieve partition 12\n",
"Will retrieve partition 13\n",
"Will retrieve partition 14\n",
"Will retrieve partition 15\n",
"Will retrieve partition 16\n",
"Will retrieve partition 17\n",
"Will retrieve partition 18\n",
"Will retrieve partition 19\n",
"Will retrieve partition 20\n",
"Will retrieve partition 21\n",
"Will retrieve partition 22\n",
"Will retrieve partition 23\n",
"Will retrieve partition 24\n",
"Will retrieve partition 25\n",
"Will retrieve partition 26\n",
"Will retrieve partition 27\n",
"Will retrieve partition 28\n",
"Will retrieve partition 29\n",
"Will retrieve partition 30\n",
"Will retrieve partition 31\n",
"Will retrieve partition 32\n",
"Will retrieve partition 33\n",
"Will retrieve partition 34\n",
"Will retrieve partition 35\n",
"Will retrieve partition 36\n",
"Will retrieve partition 37\n",
"Will retrieve partition 38\n",
"Will retrieve partition 39\n",
"Will retrieve partition 40\n",
"Will retrieve partition 41\n",
"Will retrieve partition 42\n",
"Will retrieve partition 43\n",
"Will retrieve partition 44\n",
"Will retrieve partition 45\n",
"Will retrieve partition 46\n",
"Will retrieve partition 47\n",
"Will retrieve partition 48\n",
"Will retrieve partition 49\n"
]
},
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
Dump daily one-hot concept data to dolphindb.
3 years ago
" <th>code</th>\n",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
Dump daily one-hot concept data to dolphindb.
3 years ago
" <td>000400.SZ</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>000558.SZ</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>000602.SZ</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>000677.SZ</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>000731.SZ</td>\n",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
Dump daily one-hot concept data to dolphindb.
3 years ago
" code\n",
"0 000400.SZ\n",
"1 000558.SZ\n",
"2 000602.SZ\n",
"3 000677.SZ\n",
"4 000731.SZ"
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
]
},
Dump daily one-hot concept data to dolphindb.
3 years ago
"execution_count": 9,
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
Dump daily one-hot concept data to dolphindb.
3 years ago
"import pandas as pd\n",
"\n",
"df_list = []\n",
"for i in range(50):\n",
" print(f\"Will retrieve partition {i}\")\n",
" df_list.append(\n",
" sess.run(f\"\"\"\n",
" select code from tbl where partition(code, {i}) group by code;\n",
" \"\"\")\n",
" )\n",
"df = pd.concat(df_list, axis=0)\n",
"df.head()"
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
]
},
{
"cell_type": "code",
Dump daily one-hot concept data to dolphindb.
3 years ago
"execution_count": 26,
"id": "10eb9af7-739f-4b77-aa92-9abfba875370",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
Dump daily one-hot concept data to dolphindb.
3 years ago
" <th>code</th>\n",
" <th>m_nDate</th>\n",
" <th>cnt</th>\n",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
Dump daily one-hot concept data to dolphindb.
3 years ago
" <td>689009.SH</td>\n",
" <td>2021-08-19</td>\n",
" <td>2728</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>689009.SH</td>\n",
" <td>2022-02-16</td>\n",
" <td>2570</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>689009.SH</td>\n",
" <td>2022-03-02</td>\n",
" <td>2428</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>689009.SH</td>\n",
" <td>2022-03-31</td>\n",
" <td>2584</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>689009.SH</td>\n",
" <td>2022-04-08</td>\n",
" <td>2872</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>689009.SH</td>\n",
" <td>2022-04-12</td>\n",
" <td>2902</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>689009.SH</td>\n",
" <td>2022-04-18</td>\n",
" <td>2766</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>689009.SH</td>\n",
" <td>2022-04-20</td>\n",
" <td>2584</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>689009.SH</td>\n",
" <td>2022-04-22</td>\n",
" <td>2858</td>\n",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
Dump daily one-hot concept data to dolphindb.
3 years ago
" code m_nDate cnt\n",
"0 689009.SH 2021-08-19 2728\n",
"1 689009.SH 2022-02-16 2570\n",
"2 689009.SH 2022-03-02 2428\n",
"3 689009.SH 2022-03-31 2584\n",
"4 689009.SH 2022-04-08 2872\n",
"5 689009.SH 2022-04-12 2902\n",
"6 689009.SH 2022-04-18 2766\n",
"7 689009.SH 2022-04-20 2584\n",
"8 689009.SH 2022-04-22 2858"
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
]
},
Dump daily one-hot concept data to dolphindb.
3 years ago
"execution_count": 26,
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
"metadata": {},
"output_type": "execute_result"
}
],
Dump daily one-hot concept data to dolphindb.
3 years ago
"source": [
"sess.run(\"\"\"\n",
" tbl = loadTable(\"dfs://hft_stock_ts\", 'TickQueuePartitioned');\n",
" select code, m_nDate, count(*) as cnt from tbl where code='689009.SH' and count(*) < 3000 group by code, m_nDate map;\n",
"\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "09332a23-4c1b-4d9f-8e7b-c7ef74611700",
"metadata": {},
"outputs": [],
"source": [
"df.sort_values('code').to_csv('../assets/TickQueuePartitioned.log.csv')"
]
},
{
"cell_type": "code",
"execution_count": 136,
"id": "4ba45027-bbb5-4b27-99da-3452cc8d2f1c",
"metadata": {},
"outputs": [
{
"ename": "RuntimeError",
"evalue": "<Exception> in run: Couldn't send script/function to the remote host because the connection has been closed",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mRuntimeError\u001b[0m Traceback (most recent call last)",
"Input \u001b[0;32mIn [136]\u001b[0m, in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43msess\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrun\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m\"\"\"\u001b[39;49m\n\u001b[1;32m 2\u001b[0m \u001b[38;5;124;43m select count(*) from (\u001b[39;49m\n\u001b[1;32m 3\u001b[0m \u001b[38;5;124;43m select code, m_nDate, count(*) as cnt from tbl where code=\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43m300065.SZ\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43m group by code, m_nDate map\u001b[39;49m\n\u001b[1;32m 4\u001b[0m \u001b[38;5;124;43m );\u001b[39;49m\n\u001b[1;32m 5\u001b[0m \u001b[38;5;124;43m\"\"\"\u001b[39;49m\u001b[43m)\u001b[49m\n",
"File \u001b[0;32m~/.venv/tinysoft/lib/python3.8/site-packages/dolphindb/session.py:161\u001b[0m, in \u001b[0;36msession.run\u001b[0;34m(self, script, *args, **kwargs)\u001b[0m\n\u001b[1;32m 159\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mfetchSize\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;129;01min\u001b[39;00m kwargs\u001b[38;5;241m.\u001b[39mkeys():\n\u001b[1;32m 160\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m BlockReader(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcpp\u001b[38;5;241m.\u001b[39mrunBlock(script, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs))\n\u001b[0;32m--> 161\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcpp\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrun\u001b[49m\u001b[43m(\u001b[49m\u001b[43mscript\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
"\u001b[0;31mRuntimeError\u001b[0m: <Exception> in run: Couldn't send script/function to the remote host because the connection has been closed"
]
}
],
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
"source": [
"sess.run(\"\"\"\n",
" select count(*) from (\n",
Dump daily one-hot concept data to dolphindb.
3 years ago
" select code, m_nDate, count(*) as cnt from tbl where code='300065.SZ' group by code, m_nDate map\n",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
" );\n",
reorganize the directory structure
3 years ago
"\"\"\")"
]
},
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
{
"cell_type": "code",
Dump daily one-hot concept data to dolphindb.
3 years ago
"execution_count": 126,
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
"id": "29ab8af5-e571-4064-b691-a186d9fb4d08",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
Update README.md
3 years ago
" <th>count</th>\n",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
" </tr>\n",
" <tr>\n",
" <th>m_nDate</th>\n",
Update README.md
3 years ago
" <th></th>\n",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
Dump daily one-hot concept data to dolphindb.
3 years ago
"Empty DataFrame\n",
"Columns: [count]\n",
"Index: []"
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
]
},
Dump daily one-hot concept data to dolphindb.
3 years ago
"execution_count": 126,
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df = sess.run(\"\"\"\n",
Dump daily one-hot concept data to dolphindb.
3 years ago
" select m_nDate, count(*) from tbl where code='300066.SZ' group by m_nDate map;\n",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
"\"\"\").set_index('m_nDate')\n",
"df"
]
},
{
"cell_type": "code",
"execution_count": 51,
"id": "013d191b-2a41-4d7b-8382-b6ee8764c2d0",
"metadata": {},
"outputs": [],
"source": [
"import sqlalchemy as sa\n",
"engine = sa.create_engine(\n",
" 'mssql+pyodbc://sa:passw0rd!@192.168.1.7/master?driver=ODBC+Driver+18+for+SQL+Server',\n",
" connect_args = {\n",
" \"TrustServerCertificate\": \"yes\"\n",
" }, echo=False)"
]
},
{
"cell_type": "code",
"execution_count": 54,
"id": "c90e8e9d-cde0-4a7a-afd3-ea771483f001",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'20130104'"
]
},
"execution_count": 54,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"with engine.connect() as conn:\n",
" rows = conn.execute(\"select * from [Level2BytesTick].dbo.[Tick] where S_INFO_WINDCODE='002182.SZ'\").fetchall()\n",
" rows = [row for row in rows]\n",
" "
]
},
{
"cell_type": "code",
"execution_count": 58,
"id": "525dfcf1-720a-4b65-90b5-28d0c2118cf5",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 58,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd\n",
"dt = pd.to_datetime(rows[0][1], format='%Y%m%d')\n",
"dt in df.index"
]
},
reorganize the directory structure
3 years ago
{
"cell_type": "code",
"execution_count": 31,
"id": "0e6e949b-ff25-40eb-a70a-72a3a6639aec",
"metadata": {},
"outputs": [
{
"ename": "RuntimeError",
"evalue": "<Exception> in run: Failed to read response header from the socket with IO error type 13",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mRuntimeError\u001b[0m Traceback (most recent call last)",
"Input \u001b[0;32mIn [31]\u001b[0m, in \u001b[0;36m<cell line: 15>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m code \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\"\"\u001b[39m\n\u001b[1;32m 2\u001b[0m \u001b[38;5;124mtbl = loadTable(\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mdfs://hft_stock_ts\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m, \u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mTickPartitioned\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m)\u001b[39m\n\u001b[1;32m 3\u001b[0m \n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 13\u001b[0m \u001b[38;5;124m}\u001b[39m\n\u001b[1;32m 14\u001b[0m \u001b[38;5;124m\"\"\"\u001b[39m\n\u001b[0;32m---> 15\u001b[0m \u001b[43msess\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrun\u001b[49m\u001b[43m(\u001b[49m\u001b[43mcode\u001b[49m\u001b[43m)\u001b[49m\n",
"File \u001b[0;32m~/.venv/tinysoft/lib/python3.8/site-packages/dolphindb/session.py:161\u001b[0m, in \u001b[0;36msession.run\u001b[0;34m(self, script, *args, **kwargs)\u001b[0m\n\u001b[1;32m 159\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mfetchSize\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;129;01min\u001b[39;00m kwargs\u001b[38;5;241m.\u001b[39mkeys():\n\u001b[1;32m 160\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m BlockReader(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcpp\u001b[38;5;241m.\u001b[39mrunBlock(script, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs))\n\u001b[0;32m--> 161\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcpp\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrun\u001b[49m\u001b[43m(\u001b[49m\u001b[43mscript\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
"\u001b[0;31mRuntimeError\u001b[0m: <Exception> in run: Failed to read response header from the socket with IO error type 13"
]
}
],
"source": [
"code = \"\"\"\n",
"tbl = loadTable(\"dfs://hft_stock_ts\", 'TickPartitioned')\n",
"\n",
"for (code in ['000666.SZ' , '000667.SZ', '000668.SZ', '000669.SZ']) {\n",
"\tdel_date = 2013.01.01\n",
"\t//end_date = 2013.01.31\n",
"\tdo {\n",
"\t\t//print('Deleting ' + code + ' from ' + start_date + ' to ' + end_date);\n",
"\t\tprint('Deleting ' + code + ' of ' + del_date);\n",
Dump daily one-hot concept data to dolphindb.
3 years ago
"\t\tdelete from tbl where m_nDate=del_date and code=code;\n",
reorganize the directory structure
3 years ago
"\t\tdel_date = temporalAdd(del_date, 1, \"d\");\n",
"\t} while (del_date <= 2022.12.31)\n",
"}\n",
"\"\"\"\n",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
"# 这段代码运行好赶紧注释掉，非常危险\n",
"# sess.run(code)"
reorganize the directory structure
3 years ago
]
},
{
"cell_type": "code",
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
"execution_count": 37,
reorganize the directory structure
3 years ago
"id": "ea0501e7-d416-45ce-add5-e443c55f158c",
"metadata": {},
1. Re-organize the directory structure, creating sub-packages for `DDBLoader`. 2. Update `DDBHFTLoader` to better handle the journal file. 3. Move the entrance scripts to `src/run.py`.
3 years ago
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>code</th>\n",
" <th>industry_code</th>\n",
" <th>industry_name</th>\n",
" <th>industry_level</th>\n",
" <th>enter_date</th>\n",
" <th>exit_date</th>\n",
" <th>is_new</th>\n",
" <th>industry_class</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>000017.SZ</td>\n",
" <td>SWHY310000</td>\n",
" <td>申万交运设备</td>\n",
" <td>1</td>\n",
" <td>1992-03-31</td>\n",
" <td>2011-10-10</td>\n",
" <td>0</td>\n",
" <td>SWHY</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>000017.SZ</td>\n",
" <td>SWHY310300</td>\n",
" <td>申万非汽车交运设备</td>\n",
" <td>2</td>\n",
" <td>1992-03-31</td>\n",
" <td>2011-10-10</td>\n",
" <td>0</td>\n",
" <td>SWHY</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>000017.SZ</td>\n",
" <td>SWHY310301</td>\n",
" <td>申万摩托车</td>\n",
" <td>3</td>\n",
" <td>1992-03-31</td>\n",
" <td>2011-10-10</td>\n",
" <td>0</td>\n",
" <td>SWHY</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>000017.SZ</td>\n",
" <td>SWHY310000</td>\n",
" <td>申万交运设备</td>\n",
" <td>1</td>\n",
" <td>2011-10-10</td>\n",
" <td>2014-02-21</td>\n",
" <td>0</td>\n",
" <td>SWHY</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>000017.SZ</td>\n",
" <td>SWHY310300</td>\n",
" <td>申万非汽车交运设备</td>\n",
" <td>2</td>\n",
" <td>2011-10-10</td>\n",
" <td>2014-02-21</td>\n",
" <td>0</td>\n",
" <td>SWHY</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>95</th>\n",
" <td>000157.SZ</td>\n",
" <td>SWHY260202</td>\n",
" <td>申万工程机械</td>\n",
" <td>3</td>\n",
" <td>2000-10-12</td>\n",
" <td>2014-02-21</td>\n",
" <td>0</td>\n",
" <td>SWHY</td>\n",
" </tr>\n",
" <tr>\n",
" <th>96</th>\n",
" <td>000157.SZ</td>\n",
" <td>SWHY640000</td>\n",
" <td>申万机械设备</td>\n",
" <td>1</td>\n",
" <td>2014-02-21</td>\n",
" <td>2021-12-13</td>\n",
" <td>0</td>\n",
" <td>SWHY</td>\n",
" </tr>\n",
" <tr>\n",
" <th>97</th>\n",
" <td>000157.SZ</td>\n",
" <td>SWHY640200</td>\n",
" <td>申万专用设备</td>\n",
" <td>2</td>\n",
" <td>2014-02-21</td>\n",
" <td>2021-12-13</td>\n",
" <td>0</td>\n",
" <td>SWHY</td>\n",
" </tr>\n",
" <tr>\n",
" <th>98</th>\n",
" <td>000157.SZ</td>\n",
" <td>SWHY640201</td>\n",
" <td>申万工程机械</td>\n",
" <td>3</td>\n",
" <td>2014-02-21</td>\n",
" <td>2021-12-13</td>\n",
" <td>0</td>\n",
" <td>SWHY</td>\n",
" </tr>\n",
" <tr>\n",
" <th>99</th>\n",
" <td>000157.SZ</td>\n",
" <td>SWHY640000</td>\n",
" <td>申万机械设备</td>\n",
" <td>1</td>\n",
" <td>2021-12-13</td>\n",
" <td>NaT</td>\n",
" <td>1</td>\n",
" <td>SWHY</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>100 rows × 8 columns</p>\n",
"</div>"
],
"text/plain": [
" code industry_code industry_name industry_level enter_date \\\n",
"0 000017.SZ SWHY310000 申万交运设备 1 1992-03-31 \n",
"1 000017.SZ SWHY310300 申万非汽车交运设备 2 1992-03-31 \n",
"2 000017.SZ SWHY310301 申万摩托车 3 1992-03-31 \n",
"3 000017.SZ SWHY310000 申万交运设备 1 2011-10-10 \n",
"4 000017.SZ SWHY310300 申万非汽车交运设备 2 2011-10-10 \n",
".. ... ... ... ... ... \n",
"95 000157.SZ SWHY260202 申万工程机械 3 2000-10-12 \n",
"96 000157.SZ SWHY640000 申万机械设备 1 2014-02-21 \n",
"97 000157.SZ SWHY640200 申万专用设备 2 2014-02-21 \n",
"98 000157.SZ SWHY640201 申万工程机械 3 2014-02-21 \n",
"99 000157.SZ SWHY640000 申万机械设备 1 2021-12-13 \n",
"\n",
" exit_date is_new industry_class \n",
"0 2011-10-10 0 SWHY \n",
"1 2011-10-10 0 SWHY \n",
"2 2011-10-10 0 SWHY \n",
"3 2014-02-21 0 SWHY \n",
"4 2014-02-21 0 SWHY \n",
".. ... ... ... \n",
"95 2014-02-21 0 SWHY \n",
"96 2021-12-13 0 SWHY \n",
"97 2021-12-13 0 SWHY \n",
"98 2021-12-13 0 SWHY \n",
"99 NaT 1 SWHY \n",
"\n",
"[100 rows x 8 columns]"
]
},
"execution_count": 37,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sess.run(\"\"\"\n",
"tbl = loadTable(\"dfs://info_stock_ts\", 'info_industry');\n",
"select top 100 * from tbl map;\n",
"\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "16dd860d-049f-420e-8d04-d1f8969e5ed1",
"metadata": {},
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
3 years ago
"outputs": [],
Dump daily one-hot concept data to dolphindb.
3 years ago
"source": [
"sess = ddb.session('192.168.1.7', 8848)\n",
"sess.login('admin', '123456')\n",
"\n",
"code = \"\"\"\n",
"tbl = loadTable(\"dfs://hft_stock_ts\", 'TickQueuePartitioned')\n",
"code = '689009.SH'\n",
"\n",
"del_date = 2013.01.01\n",
"do {\n",
" //print('Deleting ' + code + ' from ' + start_date + ' to ' + end_date);\n",
" print('Deleting ' + code + ' of ' + del_date);\n",
" delete from tbl where m_nDate=del_date and code=code;\n",
" del_date = temporalAdd(del_date, 1, \"d\");\n",
"} while (del_date <= 2022.12.31)\n",
"\"\"\"\n",
"# 这段代码运行好赶紧注释掉，非常危险\n",
"# sess.run(code)"
]
1. Add `DDBDailyFactor.py`, which creates daily factors from HFT data. 2. Correct some typo in `DDBLoader.py`, but shouldn't affect its normal behaviour.
3 years ago
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.13"
}
},
"nbformat": 4,
"nbformat_minor": 5
}