In this example we are launching 4 asynchronous extracts.
RTRs are generated from the ENTITYEXTRACT.txt query file located in testFolder:
EN|EN|ActionType; EXTRACT; StreamName; eagle_ml-2-0_default_out_q; FeedType; ENTITYEXTRACT; maxrows; 10; EN|EN|ActionType; EXTRACT; StreamName; eagle_ml-2-0_default_out_q; FeedType; ENTITYEXTRACT; maxrows; 20; EN|EN|ActionType; EXTRACT; StreamName; eagle_ml-2-0_default_out_q; FeedType; ENTITYEXTRACT; maxrows; 30; EN|EN|ActionType; EXTRACT; StreamName; eagle_ml-2-0_default_out_q; FeedType; ENTITYEXTRACT; maxrows; 40;
- Two queries are executed in parallel (-threads=2)
- Metrics are have additional logging in metricsDir folder
- Extracted data is saved to results folder
- All necessary libraries required for the process are in the lib folder
To start execution we create runapitest.cmd file with the following contents:
set EAGLEAPIPASS=eagle1 java -jar lib/eaglemlapiclient-1.0.1.jar -file=testFolder/ENTITYEXTRACT.txt -execution=async -endpoint=http://10.100.38.145/EagleMLWebService20 -login=eagleadmin -resultsdir=results -threads=2 -csvmetricsdir=metricsdir
To set IP address to receive response use (port is optional, any available port is used if value is not set):
-callback-address=clientipaddress[:port]
This is how a working folder with such parameters should look like:
Run runapitest.cmd:
Let's see what is happening:
Two RTRs have been generated from line 1 and line 2 of the query file with corrIds RM9Y4J2V5WKD and KHB2XM3SHXRB:
INFO [LINE:1,CORRID:EN_RM9Y4J2V5WKD] Executing request. INFO [LINE:2,CORRID:EN_KHB2XM3SHXRB] Executing request.
These two RTRs have been taken and sent to EJM in two separate threads:
INFO [LINE:1,CORRID:EN_RM9Y4J2V5WKD] Request execution initiated. Waiting for reply. INFO [LINE:2,CORRID:EN_KHB2XM3SHXRB] Request execution initiated. Waiting for reply.
Results of these requests arrived will SUCCESS status:
INFO [LINE:2,CORRID:EN_KHB2XM3SHXRB] Completed successfully in 27618 ms INFO [LINE:1,CORRID:EN_RM9Y4J2V5WKD] Completed successfully in 27783 ms
Extract files are saved to results folder with the following names:
EN_KHB2XM3SHXR B_ENTITYEXTRACT_20_20161208.xml.gz and
EN_RM9Y4J2V5WK D_ENTITYEXTRACT_10_20161208.xml.gz :INFO [LINE:2,CORRID:EN_KHB2XM3SHXRB] Output file results\EN_KHB2XM3SHXR B_ENTITYEXTRACT_20_20161208.xml.gz written in 2 ms INFO [LINE:1,CORRID:EN_RM9Y4J2V5WKD] Output file results\EN_RM9Y4J2V5WK D_ENTITYEXTRACT_10_20161208.xml.gz written in 2 ms
The same 4 steps are then performed for lines 3 and 4 of the testFolder/ENTITYEXTRACT.txt query file.
You can trace these processes in Message Center Console:
We can also find the RTR generated from line 2 of testFolder/ENTITYEXTRACT.txt here:
<EagleML xmlnxsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="RunTaskRequest" eaglemlVersion="2-0" eaglemlType="RunTaskRequest"> <header> <messageId>EN_KHB2XM3SHXRB</messageId> <sentBy>user</sentBy> <sendTo>http://www.eagleinvsys.com/eagle_ml-2-0_default_cm_control_message</sendTo> <replyTo>http://10.80.33.106:63851/eagleapicallback</replyTo> <creationTimestamp>2016-12-08 05:26:30</creationTimestamp> </header> <taskIdentifier> <correlationId correlationIdScheme="correlationIdScheme">EN_KHB2XM3SHXRB</correlationId> <businessTaskId correlationIdScheme="businessTaskIdScheme">EN</businessTaskId> </taskIdentifier> <synchronousExecution>N</synchronousExecution> <taskParameters> <taskParameter> <name>maxrows</name> <value>20</value> </taskParameter> <taskParameter> <name>FeedType</name> <value>ENTITYEXTRACT</value> </taskParameter> <taskParameter> <name>ActionType</name> <value>EXTRACT</value> </taskParameter> <taskParameter> <name>StreamName</name> <value>eagle_ml-2-0_default_out_q</value> </taskParameter> </taskParameters> </EagleML>
For line 1 of testFolder/ENTITYEXTRACT.txt (corrId EN_RM9Y4J2V5WKD):
<EagleML xmlnxsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="RunTaskRequest" eaglemlVersion="2-0" eaglemlType="RunTaskRequest"> <header> <messageId>EN_RM9Y4J2V5WKD</messageId> <sentBy>user</sentBy> <sendTo>http://www.eagleinvsys.com/eagle_ml-2-0_default_cm_control_message</sendTo> <replyTo>http://10.80.33.106:63851/eagleapicallback</replyTo> <creationTimestamp>2016-12-08 05:26:30</creationTimestamp> </header> <taskIdentifier> <correlationId correlationIdScheme="correlationIdScheme">EN_RM9Y4J2V5WKD</correlationId> <businessTaskId correlationIdScheme="businessTaskIdScheme">EN</businessTaskId> </taskIdentifier> <synchronousExecution>no</synchronousExecution> <taskParameters> <taskParameter> <name>maxrows</name> <value>10</value> </taskParameter> <taskParameter> <name>FeedType</name> <value>ENTITYEXTRACT</value> </taskParameter> <taskParameter> <name>ActionType</name> <value>EXTRACT</value> </taskParameter> <taskParameter> <name>StreamName</name> <value>eagle_ml-2-0_default_out_q</value> </taskParameter> </taskParameters> </EagleML>
You can see the trigger created to send via WS in the screenshot:
Now let's check the result of extraction:
results folder:
metricsDir folder:
This folder contains CSV-formatted copies of metrics:2016-12-08 17:27:11,564 [ main] metrics INFO type=HISTOGRAM, name=eagle.eaglemlapi.asyncrequest.reply.datasize, count=4, min=3124, max=9560, mean=6656.652851898487, stddev=2498.2503209346633, median=7992.0, p75=9560.0, p95=9560.0, p98=9560.0, p99=9560.0, p999=9560.0 2016-12-08 17:27:11,575 [ main] metrics INFO type=TIMER, name=eagle.eaglemlapi.asyncrequest.ack, count=4, min=433.44046499999996, max=982.969252, mean=656.0627232246051, stddev=266.7120455024304, median=443.087637, p75=982.248382, p95=982.969252, p98=982.969252, p99=982.969252, p999=982.969252, mean_rate=0.09569222510939479, m1=0.2502864350357026, m5=0.36234737878772627, m15=0.3869346046982117, rate_unit=events/second, duration_unit=milliseconds 2016-12-08 17:27:11,584 [ main] metrics INFO type=TIMER, name=eagle.eaglemlapi.asyncrequest.ack.parse, count=4, min=1.035736, max=70.099637, mean=28.030216767684493, stddev=33.028890433870245, median=1.1030609999999998, p75=66.805641, p95=70.099637, p98=70.099637, p99=70.099637, p999=70.099637, mean_rate=0.09567336277609972, m1=0.2502864350357026, m5=0.36234737878772627, m15=0.3869346046982117, rate_unit=events/second, duration_unit=milliseconds 2016-12-08 17:27:11,592 [ main] metrics INFO type=TIMER, name=eagle.eaglemlapi.asyncrequest.reply.processed, count=4, min=10765.71042, max=27779.385183, mean=19207.10520679705, stddev=7819.9637393914345, median=13408.791325, p75=27614.247354, p95=27779.385183, p98=27779.385183, p99=27779.385183, p999=27779.385183, mean_rate=0.09676284593846161, m1=0.04306349380181901, m5=0.00970037962819326, m15=0.0032996045879426063, rate_unit=events/second, duration_unit=milliseconds 2016-12-08 17:27:11,611 [ main] metrics INFO type=TIMER, name=eagle.eaglemlapi.asyncrequest.reply.received, count=4, min=10478.521084, max=27636.275748, mean=18952.941412367734, stddev=7889.477451066064, median=13113.243433, p75=27468.587370999998, p95=27636.275748, p98=27636.275748, p99=27636.275748, p999=27636.275748, mean_rate=0.09671918008435168, m1=0.04306349380181901, m5=0.00970037962819326, m15=0.0032996045879426063, rate_unit=events/second, duration_unit=milliseconds 2016-12-08 17:27:11,620 [ main] metrics INFO type=TIMER, name=eagle.eaglemlapi.asyncrequest.reply.saved, count=4, min=117.900759, max=272.801913, mean=201.4752817701053, stddev=73.60354060282215, median=264.685561, p75=272.801913, p95=272.801913, p98=272.801913, p99=272.801913, p999=272.801913, mean_rate=0.09669799624789979, m1=0.04306349380181901, m5=0.00970037962819326, m15=0.0032996045879426063, rate_unit=events/second, duration_unit=milliseconds 2016-12-08 17:27:11,629 [ main] metrics INFO type=TIMER, name=eagle.eaglemlapi.asyncrequest.reply.tsrparsed, count=4, min=6.223863, max=12.620554, mean=9.656095528348052, stddev=2.382493382046132, median=9.507185, p75=10.859021, p95=12.620554, p98=12.620554, p99=12.620554, p999=12.620554, mean_rate=0.09667671713548119, m1=0.04306349380181901, m5=0.00970037962819326, m15=0.0032996045879426063, rate_unit=events/second, duration_unit=milliseconds
eagle.eaglemlapi.asyncrequest.reply.datasize – size of incoming data;
eagle.eaglemlapi.asyncrequest.ack – time, in which ACK has arrived;
eagle.eaglemlapi.asyncrequest.ack.parse – time of ACK processing;
eagle.eaglemlapi.asyncrequest.reply.processed – time of multipart (TSR+data extact) processing;
eagle.eaglemlapi.asyncrequest.reply.received – time, in which multipart has arrived;
eagle.eaglemlapi.asyncrequest.reply.saved – time spent on saving extract data;
eagle.eaglemlapi.asyncrequest.reply.tsrparsed – time of TSR processing;