Skip to content
Projects
Groups
Snippets
Help
This project
Loading...
Sign in / Register
Toggle navigation
S
smallproject
Project
Overview
Details
Activity
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
mahaisong
smallproject
Commits
79a0baa7
Commit
79a0baa7
authored
Jun 04, 2018
by
mahaisong
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
fix:调度程序完成
parent
e8516d45
Show whitespace changes
Inline
Side-by-side
Showing
9 changed files
with
48 additions
and
28 deletions
+48
-28
.suo
...求-JSON贝格ES导入本地ES/ImportLocalES/.vs/ImportLocalES/v14/.suo
+0
-0
.suo
10.Paul需求-JSON贝格ES导入本地ES/MinderES/.vs/MinderES/v14/.suo
+0
-0
BeigeOracle_MinderES.csproj
...MinderES/BeigeOracle_MinderES/BeigeOracle_MinderES.csproj
+8
-2
ReadMe.md
...需求-JSON贝格ES导入本地ES/MinderES/BeigeOracle_MinderES/ReadMe.md
+3
-1
quartz_jobs.xml
...N贝格ES导入本地ES/MinderES/BeigeOracle_MinderES/quartz_jobs.xml
+9
-8
ESAccess.cs
...求-JSON贝格ES导入本地ES/MinderES/HTCommon/DataAccess/ESAccess.cs
+6
-6
ReadMe.md
...aul需求-JSON贝格ES导入本地ES/MinderES/TaikorES_MinderES/ReadMe.md
+6
-1
TaikorES_MinderES.csproj
...入本地ES/MinderES/TaikorES_MinderES/TaikorES_MinderES.csproj
+7
-2
quartz_jobs.xml
...JSON贝格ES导入本地ES/MinderES/TaikorES_MinderES/quartz_jobs.xml
+9
-8
No files found.
10.Paul需求-JSON贝格ES导入本地ES/ImportLocalES/.vs/ImportLocalES/v14/.suo
View file @
79a0baa7
No preview for this file type
10.Paul需求-JSON贝格ES导入本地ES/MinderES/.vs/MinderES/v14/.suo
View file @
79a0baa7
No preview for this file type
10.Paul需求-JSON贝格ES导入本地ES/MinderES/BeigeOracle_MinderES/BeigeOracle_MinderES.csproj
View file @
79a0baa7
<?xm
l version="1.0" encoding="utf-8"?>
<?xm
l version="1.0" encoding="utf-8"?>
...
@@ -15,7 +15,7 @@
...
@@ -15,7 +15,7 @@
<TargetFrameworkProfile
/>
<TargetFrameworkProfile
/>
</PropertyGroup>
</PropertyGroup>
<PropertyGroup
Condition=
" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' "
>
<PropertyGroup
Condition=
" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' "
>
<PlatformTarget>
AnyCPU
</PlatformTarget>
<PlatformTarget>
x64
</PlatformTarget>
<DebugSymbols>
true
</DebugSymbols>
<DebugSymbols>
true
</DebugSymbols>
<DebugType>
full
</DebugType>
<DebugType>
full
</DebugType>
<Optimize>
false
</Optimize>
<Optimize>
false
</Optimize>
...
@@ -104,8 +104,10 @@
...
@@ -104,8 +104,10 @@
<Reference
Include=
"System.Xml"
/>
<Reference
Include=
"System.Xml"
/>
</ItemGroup>
</ItemGroup>
<ItemGroup>
<ItemGroup>
<Compile
Include=
"BeigeNews.cs"
/>
<Compile
Include=
"Program.cs"
/>
<Compile
Include=
"Program.cs"
/>
<Compile
Include=
"Properties\AssemblyInfo.cs"
/>
<Compile
Include=
"Properties\AssemblyInfo.cs"
/>
<Compile
Include=
"QuartzJobs\BeigeESJob.cs"
/>
<Compile
Include=
"QuartzJobs\TestJob.cs"
/>
<Compile
Include=
"QuartzJobs\TestJob.cs"
/>
</ItemGroup>
</ItemGroup>
<ItemGroup>
<ItemGroup>
...
@@ -136,6 +138,10 @@
...
@@ -136,6 +138,10 @@
</ItemGroup>
</ItemGroup>
<ItemGroup
/>
<ItemGroup
/>
<ItemGroup>
<ItemGroup>
<ProjectReference
Include=
"..\HJBeigeModel\HJBeigeModel.csproj"
>
<Project>
{0AFE2959-EB8E-4D60-B1B1-AB7616EA319D}
</Project>
<Name>
HJBeigeModel
</Name>
</ProjectReference>
<ProjectReference
Include=
"..\HTCommon\HTCommon.csproj"
>
<ProjectReference
Include=
"..\HTCommon\HTCommon.csproj"
>
<Project>
{E497051E-61E1-4247-93D3-7929A8C8B7F6}
</Project>
<Project>
{E497051E-61E1-4247-93D3-7929A8C8B7F6}
</Project>
<Name>
HTCommon
</Name>
<Name>
HTCommon
</Name>
...
...
10.Paul需求-JSON贝格ES导入本地ES/MinderES/BeigeOracle_MinderES/ReadMe.md
View file @
79a0baa7
Mind
erESFromBeigeOracle
Mind
erESFromBeigeOracle
...
@@ -10,3 +10,5 @@
...
@@ -10,3 +10,5 @@
1.
在mysql数据库(minder.palas_v5.syncoffset )中存储HuaJinNews_Beige记录最后的last_itemid。
1.
在mysql数据库(minder.palas_v5.syncoffset )中存储HuaJinNews_Beige记录最后的last_itemid。
2.
每次执行时,读取mysql中最后一条last_itemid,查找大于此ID的所有值,将输入组装插入Minder ES。
10.Paul需求-JSON贝格ES导入本地ES/MinderES/BeigeOracle_MinderES/quartz_jobs.xml
View file @
79a0baa7
<?xm
l version="1.0" encoding="UTF-8"?>
<?xm
l version="1.0" encoding="UTF-8"?>
...
@@ -12,26 +12,26 @@
...
@@ -12,26 +12,26 @@
<!--begin-每5秒钟执行一次-
检查心跳状态
-->
<!--begin-每5秒钟执行一次-
BeigeESJob
-->
<job>
<job>
<name>
Cycle_
NoHeart
</name>
<name>
Cycle_
BeigeESJob
</name>
<group>
Test
</group>
<group>
Test
</group>
<description>
每5秒钟执行一次-
检查心跳状态
</description>
<description>
每5秒钟执行一次-
BeigeESJob
</description>
<job-type>
BeigeOracle_MinderES.QuartzJobs.
Test
Job,BeigeOracle_MinderES
</job-type>
<job-type>
BeigeOracle_MinderES.QuartzJobs.
BeigeES
Job,BeigeOracle_MinderES
</job-type>
<durable>
true
</durable>
<durable>
true
</durable>
<recover>
false
</recover>
<recover>
false
</recover>
</job>
</job>
<trigger>
<trigger>
<cron>
<cron>
<name>
Cycle_
NoHeart
Trigger
</name>
<name>
Cycle_
BeigeESJob
Trigger
</name>
<group>
Test
</group>
<group>
Test
</group>
<job-name>
Cycle_
NoHeart
</job-name>
<job-name>
Cycle_
BeigeESJob
</job-name>
<job-group>
Test
</job-group>
<job-group>
Test
</job-group>
<start-time>
2015-01-22T00:00:00+08:00
</start-time>
<start-time>
2015-01-22T00:00:00+08:00
</start-time>
<cron-expression>
0/1 * * * * ?
</cron-expression>
<cron-expression>
0/1 * * * * ?
</cron-expression>
</cron>
</cron>
</trigger>
</trigger>
<!--end-每5秒钟执行一次-
检查心跳状态
-->
<!--end-每5秒钟执行一次-
BeigeESJob
-->
</schedule>
</schedule>
</job-scheduling-data>
</job-scheduling-data>
\ No newline at end of file
10.Paul需求-JSON贝格ES导入本地ES/MinderES/HTCommon/DataAccess/ESAccess.cs
View file @
79a0baa7
usin
g
HooLab.Log
;
usin
g
HooLab.Log
;
...
@@ -97,12 +97,12 @@ namespace HTCommon.DataAccess
...
@@ -97,12 +97,12 @@ namespace HTCommon.DataAccess
try
try
{
{
Stopwatch
sw
=
new
Stopwatch
();
//
Stopwatch sw = new Stopwatch();
sw
.
Start
();
//
sw.Start();
result
=
client
.
Search
<
T
>(
searchQuery
);
result
=
client
.
Search
<
T
>(
searchQuery
);
sw
.
Stop
();
//
sw.Stop();
result
.
TookAsLong
.
ToString
();
//
result.TookAsLong.ToString();
Logger
.
Error
(
"查询用时"
+
result
.
TookAsLong
.
ToString
()+
"毫秒;总传输用时"
+
sw
.
ElapsedMilliseconds
.
ToString
()+
"毫秒;"
);
//
Logger.Error("查询用时"+ result.TookAsLong.ToString()+"毫秒;总传输用时"+sw.ElapsedMilliseconds.ToString()+ "毫秒;");
}
}
catch
(
Exception
e
)
catch
(
Exception
e
)
...
...
10.Paul需求-JSON贝格ES导入本地ES/MinderES/TaikorES_MinderES/ReadMe.md
View file @
79a0baa7
Mind
erESFromTaikorES
Mind
erESFromTaikorES
...
@@ -12,3 +12,8 @@
...
@@ -12,3 +12,8 @@
1.
在mysql数据库(minder.palas_v5.syncoffset )中存储HuaJinNews_TaikorES记录最后的last_itemid。
1.
在mysql数据库(minder.palas_v5.syncoffset )中存储HuaJinNews_TaikorES记录最后的last_itemid。
2.
载入4W爬虫ID,按照爬虫爬抓指定fectime 在今天0点到昨天0点之间的数据。
3.
循环完成4W爬虫之后结束。
10.Paul需求-JSON贝格ES导入本地ES/MinderES/TaikorES_MinderES/TaikorES_MinderES.csproj
View file @
79a0baa7
<?xm
l version="1.0" encoding="utf-8"?>
<?xm
l version="1.0" encoding="utf-8"?>
...
@@ -15,7 +15,7 @@
...
@@ -15,7 +15,7 @@
<TargetFrameworkProfile
/>
<TargetFrameworkProfile
/>
</PropertyGroup>
</PropertyGroup>
<PropertyGroup
Condition=
" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' "
>
<PropertyGroup
Condition=
" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' "
>
<PlatformTarget>
AnyCPU
</PlatformTarget>
<PlatformTarget>
x64
</PlatformTarget>
<DebugSymbols>
true
</DebugSymbols>
<DebugSymbols>
true
</DebugSymbols>
<DebugType>
full
</DebugType>
<DebugType>
full
</DebugType>
<Optimize>
false
</Optimize>
<Optimize>
false
</Optimize>
...
@@ -78,6 +78,7 @@
...
@@ -78,6 +78,7 @@
<ItemGroup>
<ItemGroup>
<Compile
Include=
"Program.cs"
/>
<Compile
Include=
"Program.cs"
/>
<Compile
Include=
"Properties\AssemblyInfo.cs"
/>
<Compile
Include=
"Properties\AssemblyInfo.cs"
/>
<Compile
Include=
"QuartzJobs\TaikorES.cs"
/>
<Compile
Include=
"QuartzJobs\TestJob.cs"
/>
<Compile
Include=
"QuartzJobs\TestJob.cs"
/>
</ItemGroup>
</ItemGroup>
<ItemGroup>
<ItemGroup>
...
@@ -91,6 +92,9 @@
...
@@ -91,6 +92,9 @@
<None
Include=
"job_scheduling_data_2_0.xsd"
>
<None
Include=
"job_scheduling_data_2_0.xsd"
>
<SubType>
Designer
</SubType>
<SubType>
Designer
</SubType>
</None>
</None>
<None
Include=
"MatchCrawlID.csv"
>
<CopyToOutputDirectory>
PreserveNewest
</CopyToOutputDirectory>
</None>
<None
Include=
"packages.config"
/>
<None
Include=
"packages.config"
/>
<None
Include=
"quartz.config"
>
<None
Include=
"quartz.config"
>
<CopyToOutputDirectory>
PreserveNewest
</CopyToOutputDirectory>
<CopyToOutputDirectory>
PreserveNewest
</CopyToOutputDirectory>
...
@@ -102,6 +106,7 @@
...
@@ -102,6 +106,7 @@
<ItemGroup>
<ItemGroup>
<Content
Include=
"quartz_jobs.xml"
>
<Content
Include=
"quartz_jobs.xml"
>
<CopyToOutputDirectory>
PreserveNewest
</CopyToOutputDirectory>
<CopyToOutputDirectory>
PreserveNewest
</CopyToOutputDirectory>
<SubType>
Designer
</SubType>
</Content>
</Content>
</ItemGroup>
</ItemGroup>
<ItemGroup>
<ItemGroup>
...
...
10.Paul需求-JSON贝格ES导入本地ES/MinderES/TaikorES_MinderES/quartz_jobs.xml
View file @
79a0baa7
<?xm
l version="1.0" encoding="UTF-8"?>
<?xm
l version="1.0" encoding="UTF-8"?>
...
@@ -12,26 +12,26 @@
...
@@ -12,26 +12,26 @@
<!--begin-每5秒钟执行一次-
检查心跳状态
-->
<!--begin-每5秒钟执行一次-
TaikorES
-->
<job>
<job>
<name>
Cycle_
NoHeart
</name>
<name>
Cycle_
TaikorES
</name>
<group>
Test
</group>
<group>
Test
</group>
<description>
每5秒钟执行一次-
检查心跳状态
</description>
<description>
每5秒钟执行一次-
TaikorES
</description>
<job-type>
TaikorES_MinderES.QuartzJobs.T
estJob
,TaikorES_MinderES
</job-type>
<job-type>
TaikorES_MinderES.QuartzJobs.T
aikorES
,TaikorES_MinderES
</job-type>
<durable>
true
</durable>
<durable>
true
</durable>
<recover>
false
</recover>
<recover>
false
</recover>
</job>
</job>
<trigger>
<trigger>
<cron>
<cron>
<name>
Cycle_
NoHeart
Trigger
</name>
<name>
Cycle_
TaikorES
Trigger
</name>
<group>
Test
</group>
<group>
Test
</group>
<job-name>
Cycle_
NoHeart
</job-name>
<job-name>
Cycle_
TaikorES
</job-name>
<job-group>
Test
</job-group>
<job-group>
Test
</job-group>
<start-time>
2015-01-22T00:00:00+08:00
</start-time>
<start-time>
2015-01-22T00:00:00+08:00
</start-time>
<cron-expression>
0/5 * * * * ?
</cron-expression>
<cron-expression>
0/5 * * * * ?
</cron-expression>
</cron>
</cron>
</trigger>
</trigger>
<!--end-每5秒钟执行一次-
检查心跳状态
-->
<!--end-每5秒钟执行一次-
TaikorES
-->
</schedule>
</schedule>
</job-scheduling-data>
</job-scheduling-data>
\ No newline at end of file
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment