EagleML API Client - Example of Use (Async)

Owned by Natalie Gesin (Deactivated)
Last updated: Aug 19, 2020 by Dmitry Donosiyan (Unlicensed)
4 min read

In this example we are launching 4 asynchronous extracts.

  • RTRs are generated from the ENTITYEXTRACT.txt query file located in testFolder:

    EN|EN|ActionType; EXTRACT; StreamName; eagle_ml-2-0_default_out_q; FeedType; ENTITYEXTRACT; maxrows; 10;
EN|EN|ActionType; EXTRACT; StreamName; eagle_ml-2-0_default_out_q; FeedType; ENTITYEXTRACT; maxrows; 20;
EN|EN|ActionType; EXTRACT; StreamName; eagle_ml-2-0_default_out_q; FeedType; ENTITYEXTRACT; maxrows; 30;
EN|EN|ActionType; EXTRACT; StreamName; eagle_ml-2-0_default_out_q; FeedType; ENTITYEXTRACT; maxrows; 40;
  • Two queries are executed in parallel (-threads=2)
  • Metrics are have additional logging in metricsDir folder
  • Extracted data is saved to results folder
  • All necessary libraries required for the process are in the lib folder

  • To start execution we create runapitest.cmd file with the following contents:

    set EAGLEAPIPASS=eagle1
java -jar lib/eaglemlapiclient-1.0.1.jar -file=testFolder/ENTITYEXTRACT.txt -execution=async -endpoint=http://10.100.38.145/EagleMLWebService20 -login=eagleadmin -resultsdir=results -threads=2 -csvmetricsdir=metricsdir

  • To set IP address to receive response use (port is optional, any available port is used if value is not set):

    -callback-address=clientipaddress[:port]

This is how a working folder with such parameters should look like:

Run runapitest.cmd:

Let's see what is happening:

  1. Two RTRs have been generated from line 1 and line 2 of the query file with corrIds RM9Y4J2V5WKD and KHB2XM3SHXRB:

    INFO  [LINE:1,CORRID:EN_RM9Y4J2V5WKD] Executing request.​
INFO  [LINE:2,CORRID:EN_KHB2XM3SHXRB] Executing request.

  2. These two RTRs have been taken and sent to EJM in two separate threads:

    INFO  [LINE:1,CORRID:EN_RM9Y4J2V5WKD] Request execution initiated. Waiting for reply.
INFO  [LINE:2,CORRID:EN_KHB2XM3SHXRB] Request execution initiated. Waiting for reply.

  3. Results of these requests arrived will SUCCESS status:

    INFO [LINE:2,CORRID:EN_KHB2XM3SHXRB] Completed successfully in 27618 ms
INFO [LINE:1,CORRID:EN_RM9Y4J2V5WKD] Completed successfully in 27783 ms

  4. Extract files are saved to results folder with the following names:
    EN_KHB2XM3SHXR B_ENTITYEXTRACT_20_20161208.xml.gz and
    EN_RM9Y4J2V5WK D_ENTITYEXTRACT_10_20161208.xml.gz :

    INFO  [LINE:2,CORRID:EN_KHB2XM3SHXRB] Output file results\EN_KHB2XM3SHXR B_ENTITYEXTRACT_20_20161208.xml.gz written in 2 ms
INFO  [LINE:1,CORRID:EN_RM9Y4J2V5WKD] Output file results\EN_RM9Y4J2V5WK D_ENTITYEXTRACT_10_20161208.xml.gz written in 2 ms

  5. The same 4 steps are then performed for lines 3 and 4 of the testFolder/ENTITYEXTRACT.txt query file.

  6. You can trace these processes in Message Center Console:

  7. We can also find the RTR generated from line 2 of testFolder/ENTITYEXTRACT.txt here:

    <EagleML xmlnxsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="RunTaskRequest" eaglemlVersion="2-0" eaglemlType="RunTaskRequest">
<header>
<messageId>EN_KHB2XM3SHXRB</messageId>
<sentBy>user</sentBy>
<sendTo>http://www.eagleinvsys.com/eagle_ml-2-0_default_cm_control_message</sendTo>
<replyTo>http://10.80.33.106:63851/eagleapicallback</replyTo>
<creationTimestamp>2016-12-08 05:26:30</creationTimestamp>
</header>
<taskIdentifier>
<correlationId correlationIdScheme="correlationIdScheme">EN_KHB2XM3SHXRB</correlationId>
<businessTaskId correlationIdScheme="businessTaskIdScheme">EN</businessTaskId>
</taskIdentifier>
<synchronousExecution>N</synchronousExecution>
<taskParameters>
<taskParameter>
<name>maxrows</name>
<value>20</value>
</taskParameter>
<taskParameter>
<name>FeedType</name>
<value>ENTITYEXTRACT</value>
</taskParameter>
<taskParameter>
<name>ActionType</name>
<value>EXTRACT</value>
</taskParameter>
<taskParameter>
<name>StreamName</name>
<value>eagle_ml-2-0_default_out_q</value>
</taskParameter>
</taskParameters>
</EagleML>


  8. For line 1 of testFolder/ENTITYEXTRACT.txt (corrId EN_RM9Y4J2V5WKD):

    <EagleML xmlnxsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="RunTaskRequest" eaglemlVersion="2-0" eaglemlType="RunTaskRequest">
<header>
<messageId>EN_RM9Y4J2V5WKD</messageId>
<sentBy>user</sentBy>
<sendTo>http://www.eagleinvsys.com/eagle_ml-2-0_default_cm_control_message</sendTo>
<replyTo>http://10.80.33.106:63851/eagleapicallback</replyTo>
<creationTimestamp>2016-12-08 05:26:30</creationTimestamp>
</header>
<taskIdentifier>
<correlationId correlationIdScheme="correlationIdScheme">EN_RM9Y4J2V5WKD</correlationId>
<businessTaskId correlationIdScheme="businessTaskIdScheme">EN</businessTaskId>
</taskIdentifier>
<synchronousExecution>N</synchronousExecution>
<taskParameters>
<taskParameter>
<name>maxrows</name>
<value>10</value>
</taskParameter>
<taskParameter>
<name>FeedType</name>
<value>ENTITYEXTRACT</value>
</taskParameter>
<taskParameter>
<name>ActionType</name>
<value>EXTRACT</value>
</taskParameter>
<taskParameter>
<name>StreamName</name>
<value>eagle_ml-2-0_default_out_q</value>
</taskParameter>
</taskParameters>
</EagleML>


  9. You can see the trigger created to send via WS in the screenshot:

  10. Now let's check the result of extraction:
    results folder:

    metricsDir folder:

    This folder contains CSV-formatted copies of metrics:

    2016-12-08 17:27:11,564 [                          main] metrics                        INFO  type=HISTOGRAM, name=eagle.eaglemlapi.asyncrequest.reply.datasize, count=4, min=3124, max=9560, mean=6656.652851898487, stddev=2498.2503209346633, median=7992.0, p75=9560.0, p95=9560.0, p98=9560.0, p99=9560.0, p999=9560.0

2016-12-08 17:27:11,575 [                          main] metrics                        INFO  type=TIMER, name=eagle.eaglemlapi.asyncrequest.ack, count=4, min=433.44046499999996, max=982.969252, mean=656.0627232246051, stddev=266.7120455024304, median=443.087637, p75=982.248382, p95=982.969252, p98=982.969252, p99=982.969252, p999=982.969252, mean_rate=0.09569222510939479, m1=0.2502864350357026, m5=0.36234737878772627, m15=0.3869346046982117, rate_unit=events/second, duration_unit=milliseconds

2016-12-08 17:27:11,584 [                          main] metrics                        INFO  type=TIMER, name=eagle.eaglemlapi.asyncrequest.ack.parse, count=4, min=1.035736, max=70.099637, mean=28.030216767684493, stddev=33.028890433870245, median=1.1030609999999998, p75=66.805641, p95=70.099637, p98=70.099637, p99=70.099637, p999=70.099637, mean_rate=0.09567336277609972, m1=0.2502864350357026, m5=0.36234737878772627, m15=0.3869346046982117, rate_unit=events/second, duration_unit=milliseconds

2016-12-08 17:27:11,592 [                          main] metrics                        INFO  type=TIMER, name=eagle.eaglemlapi.asyncrequest.reply.processed, count=4, min=10765.71042, max=27779.385183, mean=19207.10520679705, stddev=7819.9637393914345, median=13408.791325, p75=27614.247354, p95=27779.385183, p98=27779.385183, p99=27779.385183, p999=27779.385183, mean_rate=0.09676284593846161, m1=0.04306349380181901, m5=0.00970037962819326, m15=0.0032996045879426063, rate_unit=events/second, duration_unit=milliseconds

2016-12-08 17:27:11,611 [                          main] metrics                        INFO  type=TIMER, name=eagle.eaglemlapi.asyncrequest.reply.received, count=4, min=10478.521084, max=27636.275748, mean=18952.941412367734, stddev=7889.477451066064, median=13113.243433, p75=27468.587370999998, p95=27636.275748, p98=27636.275748, p99=27636.275748, p999=27636.275748, mean_rate=0.09671918008435168, m1=0.04306349380181901, m5=0.00970037962819326, m15=0.0032996045879426063, rate_unit=events/second, duration_unit=milliseconds

2016-12-08 17:27:11,620 [                          main] metrics                        INFO  type=TIMER, name=eagle.eaglemlapi.asyncrequest.reply.saved, count=4, min=117.900759, max=272.801913, mean=201.4752817701053, stddev=73.60354060282215, median=264.685561, p75=272.801913, p95=272.801913, p98=272.801913, p99=272.801913, p999=272.801913, mean_rate=0.09669799624789979, m1=0.04306349380181901, m5=0.00970037962819326, m15=0.0032996045879426063, rate_unit=events/second, duration_unit=milliseconds

2016-12-08 17:27:11,629 [                          main] metrics                        INFO  type=TIMER, name=eagle.eaglemlapi.asyncrequest.reply.tsrparsed, count=4, min=6.223863, max=12.620554, mean=9.656095528348052, stddev=2.382493382046132, median=9.507185, p75=10.859021, p95=12.620554, p98=12.620554, p99=12.620554, p999=12.620554, mean_rate=0.09667671713548119, m1=0.04306349380181901, m5=0.00970037962819326, m15=0.0032996045879426063, rate_unit=events/second, duration_unit=milliseconds

    eagle.eaglemlapi.asyncrequest.reply.datasize – size of incoming data;
    eagle.eaglemlapi.asyncrequest.ack – time, in which ACK has arrived;
    eagle.eaglemlapi.asyncrequest.ack.parse – time of ACK processing;
    eagle.eaglemlapi.asyncrequest.reply.processed – time of multipart (TSR+data extact) processing;
    eagle.eaglemlapi.asyncrequest.reply.received – time, in which multipart has arrived;
    eagle.eaglemlapi.asyncrequest.reply.saved – time spent on saving extract data;
    eagle.eaglemlapi.asyncrequest.reply.tsrparsed – time of TSR processing;