Three member brokers of National Stock Exchange of India Ltd (NSE) got secondary server access and received maximum ticks due to lack of defined policies and procedures by the Exchange, reveals a forensic review.
In the audit, Ernst & Young LLP (India) or EY, also pointed out multiple logins to a single server through multiple internet protocol (IP) addresses, allowed by four member team of NSE.
"Analysis of batches received by members indicated that 92% of ticks were received first by members connected on ports of secondary server. We were also informed by NSE that secondary POP was always active for members to make connections and same TBT market data feed was disseminated to all the POPs including secondary POP. Based on PDC logs we noted that secondary POP had connected to PDC on each trading day after it was operationalized in February 2012. The top three members who connected to secondary server were Adroit Financial Services Pvt Ltd (461 days), Parwati Capital Market Pvt Ltd (441 days) and IKM Investors Pvt Ltd (421 days)," EY says in the report.
EY says, "We further noted that there were 21 members who connected only to the secondary server and not to any of the primary servers on at least one of the trading days. Below is the list of top 10 members who logged in only to secondary server on at least one of the trading days."
NSE provided their members a colocation (Colo), which is a paid facility for doing Direct Market Access (DMA) and Algorithmic Trading (Algo trading) within its premises at Bandra Kurla Complex in Mumbai. NSE offers Colo services across market segments, which are Cash Market (CM), Currency Derivative (CD), Interest Rate Futures (IRF) and Futures & Options (F&O) segments. Except those mentioned in the ‘NSE Colocation Guidelines’ released on 16 April 2012, the Exchange had not defined policies and procedures around secondary server access.
Through this Colo service, EY says members were able to receive real time tick-by-tick (TBT) market data broadcast. Dissemination of such TBT was done through transmission control protocol/ internet protocol (TCP/IP) in NSE architecture till 30 November 2016. The same TBT data was also shared through a multicast architecture since 7 April 2014.
NSE had appointed EY to conduct a forensic review of certain allegations around colocation facility provided by NSE in the currency derivatives and interest rate futures segments. EY reviewed TCP/IP tick-by-tick architecture, available process and policy documents, emails and electronic documents of relevant employees and relevant logs of dissemination servers that were operational from 1 January 2010 to 31 December 2015 at NSE. It submitted its report on 3 November 2017 to NSE Board. Later it held discussions with Securities and Exchange Board of India (SEBI)'s Technical Advisory Committee (TAC), which led to certain additional checks. EY completed the work and submitted its report on 18 May 2018.
At NSE, there were three point-of-presence (POP) servers, including secondary with two ports each and consequently six independent dissemination queues. A member would need to be first on all the six ports across three POPs to be disseminated all the batches first on that trading day.
"Based on analysis of timestamp of batches received by member IPs, it was noted that around 28% to 55% and about 75% to 98% of batches were received first by members who had logged in first on respective ports of primary server and secondary server respectively indicating that a member logging in first on a port may not receive all batches first on that port," EY says.
The forensic report highlights how in the absence of defined policies and load balancing process, four members from NSE were assigning member IPs to ports manually. It says, "Based on our discussions with NSE team, we were informed that it was a practice (no documented policy) to limit the number of maximum IP allocations to a port of a primary POP to 30. Such assignment of member IPs to ports was done manually by four members within the PSM IICS team. We were also informed that a dynamic load balancing process was not implemented which would have distributed load evenly across the ports based on the number of connections made on each trading day."
"There was no randomisation implemented on the ports. Further, there was no dynamic load balancing to evenly distribute the member connections across all available ports on each trading day. On account of the above weaknesses, NSE’s TCP/IP TBT architecture may have caused ticks to be disseminated earlier to members who logged in ahead of others and members who logged on less loaded server/port. However, a member would need to be logged in first on all the six ports (across three POPs) to be disseminated all the ticks first on that trading day. There was no member who logged in first on all six ports on any trading day," the Report says.
Based on the information shared by the Exchange, EY says, it appears that on four trading days in 2012 (4 May 2012, 18 May 2012, 7 June 2012 and 8 June 2012), NSE monitored connections to secondary server (for CD/IRF segment) and also communicated to members that they should not be connecting without intimation by exchange and later also termed such connections as an ‘offense’. "During discussions, NSE informed us that such limited monitoring on secondary server connections in 2012 was carried out at the time of TBT infrastructure migration to colocation datacentre and till the operations were stabilised. Further, NSE informed us that they did not monitor secondary server connections after 2012," it says.
The forensic audit also found out how members were using multiple logins to a single dissemination server through multiple IPs. It says, "Based on the TBT IP allotment process as explained by NSE, there was no restriction based on any policy or procedure on a member to avail multiple TBT IPs. Similarly, multiple TBT IPs of a member could have been configured on the same port of the same dissemination server. Such configurations were done manually by a four member team of production support and management (PSM) IICS team based on the number of connections made on the ports of the primary server on that trading day."
Based on the review of login logs, EY says it observed that there were 36 members out of 58 members since operationalisation of secondary server, who had accessed both the primary servers (POP11 and POP13) at least once. Out of remaining 22 members who accessed either of the primary servers, 15 members had only one TBT IP.
"While no member logged in first and second and third on all the ports, there were 48 members out of 59 members who had logged in first on at least one of the trading days on either of the ports in the review period. There were four members, who had logged in first, second and third on one of the six ports on at least one trading day. Adroit Financial Services had logged in first and second and third on same port for 43 days (4% of the review period), which is maximum for a member," the review says.
"However," EY says, "three members, Parwati Capital Market, Adroit Financial Services and Mansukh Securities & Finance Ltd logged in first on either of the ports for more than a quarter of the review period or for more than 275 days out of total 1,103 trading days."
The main reason for this, according to the forensic review, is lack of randomisation in TCP/IP TBT architecture at NSE. “Our review of the available source code of the Primary Data Centre (PDC) and POP and discussions with NSE personnel indicates that randomisation was not implemented. In the absence of a randomizer, dissemination on each port of a TBT server was sequential based on login time of a member,” EY says in its report.
