Thursday, June 26, 2008
OCTOBER was cruising towards its close. The clock was viciously steering towards the midnight stroke. Sweat-swathed eyebrows, nervous eyeballs, twitching palms, pounding heartbeats, crossed fingers, distraught glances and agitated minds. The air was bubbling with tension, anticipation, fears, prayers and tumultuous hope. An unknown campus in Pune's Hinjewadi outskirts was literally living a rocket launch NASA moment and just a hair distance from making history. It was nothing short in stature or excitement though. The 90 per cent run of India's supercomputer-in-making was successfully through, but the clincher was actually now. The last 10 per cent run-time. It might happen, it might not. Some kilometers away on his way and continuously on the phone, Dr. N Seetha Ram Krishna, project manager, CRL and one of the key architects, understandably still kept arming his team against the Murphy's ways, "It may fail, be prepared for everything." As the reverse countdown begin, every heart and hope in the jitter-packed room started racing high. Five, four, three, two, one and … YES! The supercomputer hit the 117.9 teraflop mark. At about 11 pm on October 31 at a TAT facility in Pune's Hinjewadi IT park outskirts at CRL, shrieks of joy, sighs of achievement, and euphoria was all that could be heard, seen and felt next. India's technological razor had made its sharpest cut again. The dream was finally alive. And one hour later when Dr Krishna looked around the same room he met another once-in-a-lifetime sight.
Exhausted with 22 hours of grinding toil for the past six months and worn out of a peak of excitement and tension just some minutes back, everyone in the same room dozed off into a blithe and well-accomplished sleep. "That's a lifetime experience." It surely was. India through CRL (Computational Research Labs), a Tata Sons' wholly owned subsidiary, had claimed its space in the world top 10 supercomputer league. Fourth in the global ranking and fastest in Asia. The 120 teraflop (sustained rating) supercomputer with a peak hit of 172 teraflop was actually a reality in October 2007, with the added pride of being the largest privately funded supercomputer in the world. And the story of this distinctive feat starts six months back? May be not. India's supercomputing lineage Supercomputing may have turned as spruce, breakneck, and swanky a game as F-1 today. But turn back the leaves of history and you find all the flavors of a long, patient, and sole marathon.
Performance was at the core then too but accompanied with stamina, persistence and time. Interestingly, the annals have their origins in the late 1980s when apparently India began developing supercomputers after being refused a Cray supercomputer by the US. For India, in particular, the epic of supercomputing scribbled its first page with the Param supercomputer programme. Param, the first Indian supercomputer, was developed by the Center for Developed and Advanced Computing (C-DAC). C- DAC pioneered the supercomputing movement in the 1990's giving India her first indigenous supercomputer in 1991 (PARAM 8000). The PARAM stock continued with PARAM 8000-600, PARAM 9000, PARAM open frame, PARAM 10000 (with 100 gigaflop (floating point operations per second) and finally PARAM-PADMA in 2003. Param Padma teraflop supercomputer with one teraflop of power (a tenfold increase over the country's previous supercomputer), heralded India's entry in the top-500 league. It was ranked 171 in a list of the world's most powerful supercomputers by Top 500, a respected rating agency for the high-end computing fraternity. Another member of India's elite circuit was Kabru, a cluster of smaller computers done at the Chennai-based Institute of Mathematical Sciences (IMS) that crossed the teraflop barrier. Other stellar names of the crème-de-la-crème are the NAL's Flosolver (National Aerospace Laboratory), Anupam (Bhabha Atomic Research Center), PACE (Advanced Numerical Research Group) and CHIPPS (from the Center for Development of Telematics — CDOT, Bangalore) and the supercomputer at the Institute of Genomics and Integrative Biology, New Delhi, which finds a place among the top 500 supercomputers of the world. That was the pedigree that supercomputing had to take over from as it moved on to Eka. And interestingly, in a privately-funded environment this time. The epic entered a new epoch.
The rules, the tracks, the gear, the pitstop, everything was new, faster, and more competitive this time. And the chequered flag was only six months away. Eka incubates Tata's HPC (High Performance Computing) initiative dates back to June 2006 with the aim of becoming the one-stop-shop and achieving the iconic journey from atoms to applications. It armed CRL, its subsidiary, with the mandate of the Eka (Sanskrit for the number One) Dream. This 75-member team, which was divided into hardware, system software and applications, had beyond the obvious challenge of achieving the supercomputing power as set, also the nigh-impossible goalpost of doing all that in flat six months. It had Dr Sunil Sherlekar, the Head of R&D in CRL and also one of the founders Dr Narendra Karmarkar. This concept was presented to the Tata Sons Board to get the funding. Since then Dr Karmakar left and Dr Sherlekar stayed as the remaining founder. Incidentally, Eka also claims the distinction of being the only supercomputer funded by a corporate. CRL had the task of fully integrating and designing Eka with an in-house developed technology. The race flagged off. In June 2007 when the building infrastructure was set ready in Hinjewadi, a 4000 sq ft floor area set in record time, becomes the data center to house Eka.August saw the initial machines with 16 teraflop peaks and the first prototype going operational. September, the building blocks get ordered, set in place in due time and October sees the 172 teraflop peak system operation. In six weeks record time, the actual 120 teraflops (i.e. or trillions of calculations per second performance) happen, and Eka is born.
Nuts, bolts and paraphernalia
Eka's mandates and appetite on the technical depth were not simple by any account. Among the many objectives for CRL, was the ability to accommodate scalable parallel storage and scale at the lowest footprint space, HVAC for air distribution and cooling optimization, energy optimized operations, usage of indigenous building blocks, fully-automated monitoring and control, and accommodation of multi-system architecture and networks. Eka is built with 1794 blade servers using common off-the-shelf hardware using quad-core Intel clovertown processors. It has 400 ton cooling capacity and 2.5 MW power requirement. Its benchmark is 117.9 teraflop, and achieved final performance of 120 teraflop on a sustained basis with 172 teraflop as the peak score. There are 28 Terabytes of memory with a storage counterpart of 80 terabytes. Eka used DDR 920 gbit/sec)Infiband interconnect .technology for interconnecting the overall processors and storage, Linux OS, XC3 3.2.1 open MPI developer environment compilers and library. Cabling overall spread over 45 km falls between 10 km electrical, 15 km Infiniband and 10 km Ethernet areas. Other numbers of note include 28 TB RAM, 80 TB disk.
The revolutions in the evolution
There were quite many new approaches and dimensions during the making of Eka. The key was in the design of the architecture, then designing algorithms for application classes (a term that itself is nascent in the scientific domain and an ongoing research game), and the mapping of algorithms to the architecture. CRL also chose the not-tried-before circular layout, which serendipitously took the form of an octagon for stacking the servers and switches. Supercomputers don't come without their share of burdens. Scalability is a major issue. CRL addressed scalability by doing away with the need of connecting all nodes to all. Judicious balancing, compute and communication, and guaranteed load balancing were some of the ideas attempted here. Another concern was the cost per teraflop, which the Eka team managed to handle with reduction in interconnect switches, cables and connectors so that Interconnect scales up linearly. Similarly, usability, another issue with supercomputers was faced in the eye with innovations in library and Maths kernels thus hiding complexity of the underlying hardware, as shared by Seetha Ram Krishna. Additionally were solutions like novel interconnect architecture based on projective geometry that takes care of complexity as well as near linear speed-up of applications. There were also better algorithms for specific applications that helped Eka. In addition, the key points of uniqueness in terms of architecture are high density packing, projective geometry interconnect, hybrid parallel programming paradigm, optical interconnects and of course the circular layout as mentioned before. Props and poles As Dr. Sherlekar points out, fair share of credit of Eka goes to enabling technologies in networking, storage, power and cabling etc. While some of them had evolved to an adequate extent already, thanks to ongoing innovations, some of the path-breaking innovations that made Eka possible, in fact, happened during the making of Eka, with the concerted efforts of CRL and other partners in respective areas.
Cabling, for instance, is one major example. Eka became the testing ground of early fibercable technology since it was not feasible to use erstwhile copper cables beyond seven meters due to breakage concerns. The new technology that covered 20 meters, is already in production mode now. From processors, multi-core, programmable voltage, Interconnect Infiniband, sensors to cooling technology, every paraphernalia was as cutting-edge as Eka itself turned out to be.
120 Teraflops-milking the COW
The proof of the pudding lies in eating it. The actual work starts now when the supercomputer can start working on problems that have been waiting for the power of a superhero. Eka's usage is outlined wide and deep. Its applications - current and potential - cover a long spectrum. From system architecture research, system software research, mathematical library development, large scientific problems, application porting, optimization and development to future technology development and data center development, the possibilities have just started surfacing. Talking of simulations Eka's purview covers computational fluid dynamics and nanomaterial simulation. As Head of Applications Group Dr.Rajendra Lagu rightly says, "The young scientific minds are the ones to be looked onto now. There are for sure many applications that nobody has ever thought of yet." Eka is open for grand challenges. The exciting areas ahead range from aeroacoustics, weather modeling, carbon nanotube modeling, CFD or Computational Fluid Dynamics, Number Theory, Motif discovery, molecular docking, aircraft simulation, bio-medical simulation, to business applications like SCM (Supply Chain Management), BI (Business Intelligence), email scanning, pattern detection, video surveillance and so on.
May be the next Tata car would be completely simulated on Eka. Some 'grey' matter Eka is not without its share of criticisms and correction points. At a CSI seminar, an elderly and senior scientist in this domain pointed out areas like use of analog devices that was conspicuous by its absence in Eka; the complex use of FPGS; the perception that differential calculus is the ultimate in mathematics. CRL's explanation to them goes thus. The latter is still a topic of research and even CRL is doing work on inverse problems in this area. As to the use of analog devices, there are constraints on programmability and limits of a fixed function in an era where digital is in vogue. Even switched capacity filters, the hybrids are not in vogue, explained Dr Lagu in response.
The human in the superhuman
The real beauty behind the mystery of Eka, the supercomputer is the very absence of mystery. As Seetha Ram Krishna emphatically demystifies it, "Any company can achieve this. There's nothing complex behind it. Just some knowledge, expertise, logic, ideas and electronics put together in a system I can make it possible. "Today, with Internet as the ever-accessible ocean of knowledge and science, anyone can make their own supercomputer, albeit with levels of scale," he says. It was not a cakewalk. Every day was an ordeal, every runtime taxing and expensive. Problems were consistent, dead-ends kept coming, speed-bumpers were frequent guests, pressure levels always northwards, a new bottleneck knocked everyday and the Eka team was on an incessant 'on-the-toes' mode. But the dream, the passion, the resolve were the fuels that never ran out and made Eka a reality. The future of supercomputing is something that will unfold interestingly. "A supercomputer is a force multiplier and the third pillar of science. They are becoming all-pervasive to the games we play and the digital content we consume," says Dr Lagu. The Top500 supercomputer list would remain on Eka and CRL's dashboard. But the list, updated every six months, would be as fickle as the speed with which faster systems keep popping up. The top-ranker IBM's BlueGene/L System has achieved a benchmark of 478.2 TFLOPS. Pitted against computing superpowers like China, Japan and the US computing, India's sprint will be something to watch out for. Competition with the likes of China's Dawning, Japan's NEC's Earth Simulator and IBM's Blue Gene stirps would not be easy. The race is going to turn ruthless but will surely have India on the fast tracks.
Next on the cards for CRL, are plans of building a bigger machine that incorporates accelerators in addition to building domain-specific software libraries tuned for Eka architecture. Besides this, there are on the anvil, innovations on the software stack and productivity tools for the imminent many-core revolution. CRL also intends to explore hybrid architectural solutions for peta class machines with the use of FPGAs and accelerators. "What's a factor of concern however is the critical need of development of human resources in this space. It was tough to make about 75 to 80 member Eka team and we know how we did it. The future problems won't be the hardware or software, but the peopleware." Dr.Lagu stresses. Eka and its likes could buck the trend and drive more Indian cerebra to experience the inimitable thrill that CRL's team experienced that historical night. After all, it was indeed an eka moment. It's intriguing at this juncture then to think of some words that the father of supercomputing Seymour Cray said: "It's always easy to do the next step and it's always impossible to do two steps at a time."
(Acknowledgements to Dr Sunil Sherlakar, s the Head of R&D in CRL; Dr Rajendra Lagu, Head of Applications Group, CRL and and N Seetha Ram Krishna - project manager, CRL - all key members of Eka team and CSI –Computer Society of India's Gireendra Kasmalkar Pune for sharing the Eka story)
Vital Stats ·
1794 blade servers · 400 ton cooling capacity
· 2.5 MW power
· Benchmark performance: 117.9 teraflop (processor speed of one trillion floating point operations per second)
· Peak score: 172 teraflop
· Sustained level: 120 teraflop
· Gigs: 16 GB RAM, 80 GB hard disk
Extra-ordinary Stories of Ordinary People: 1.0
Salim ki kahani
June 25, 2008:
“ARE YOU NUTS?
Driving over 200 kmph and then spinning around the turn and halting right there at a feather’s distance to the girl! And this lanky yokel from Pune would do it?!!”, hollered the director.
“Sir, aap ek baar try to keejiye (give him a chance),” entreated the confidant spot boy. “If there’s anyone in India, who can do this stunt, it’s him,” he confidently bragged.
Ten minutes later, a car was screeching down the long road at breakneck speed, whizzing by a dazed film crew, and then taking a pronto turn, chalked two circles with the hot rubber and then suddenly took an almost impossible, suave, smooth, neat halt near the heroine.
A big applause followed and the director happily chewed back his words. After all, this was a scene long due for the past so many weeks and everyone, in the crew, from the director to the cast to the spot boys were losing their patience by now. It was one thing to steer a car at top speed but to make it saunter back in precision, and that too after some spine-chilling loops…was in one word, Phew!
So when this simple-looking hero, stepped out of the car, he was deservedly overwhelmed by hands congratulating him, thanking him - of the director, of the many assistants and of Bipasha Basu, who all, had almost given up the hope of the scene ever seeing the light of the camera.
For Salim, a common cab driver, who shuttles between Pune and Mumbai and nearby stretches ferrying passengers on normal days, this truly was not a normal day. “Maine Bipasha ke saath baith ke Thums Up piya. Unhone muje bola ki mein to ye scene ka aas hi chod chuki thi. Mera naam poocha. Accha laga. Wo din, mein bahut khush hua.” He says with a glint of the moment, he seems to line all over again as he recounts his adventures.
The sincere, sweat-earned, arcadian look on his face makes you squirm in your seat with an uneasiness. A Thums Up can be a trophy? And a small Gold Medal, can really be someone’s spring of joy forever? You have no choice but to believe as he shares his pride in an innocent, yokel but genuinely exulted voice.
“I got a gold medal for saving the life of me, the passengers on board and the bus, once when a Volvo I was driving met a brake-failure at top speed on one of the Ghats at the edgy Lonavala route. It was an absolute-death situation. And if the driver gets panicked, all is gone. Everything rests on the calmness, composure and the presence-of-mind of the man on the wheel at such a moment.” And so did he, by staying cool and smartly de-accelerating the bus by manouevring the gears. In return, he got wishes, blessings of the many passengers and a gold medal from the Travels’ proprietor.
The medal was the best and the second most emotional turn of Salim’s life.
“Mein aur meri biwi ne do din tak khana nai khaya, khusi ke mare. Uparwala bahut meherbaan hai.” Beams Salim.
Today too is one another special day in his life as he is driving me to Mumbai in an Innova. “Madam, aaj kismat se Seth ne ye ghadi chalena ko diya, kyunki doosre car nahi set ho paya. Sacchi bolta hoon, Innova drive kar ke life ne ek aur sapna poora kar diya. Majja aa gaya itna accha gaadi chalake.”
What amazes and humbles me is not the adventures this seemingly simple driver has been living off and on, but the sheer simplicity with which he embraces and celebrates life’s small but true bounties.
This was not what I expected as I opened a casual conversation on a boring three-hour drive from Pune to Mumbai one afternoon with the cab driver.
And when I ask, where he hails from, his sad but dignified and composed reply reveals the first most emotional turn of his life. Destiny brought him and his wife to Pune when an unfortunate earthquake hit his hometown in Latur. The Salim who saw his parents and other family members as mud-swathed corpses was certainly more brave than the daredevil driver he turned into. He saw, absorbed and accepted it all with a patience, faith and fortitude that probably only the common man is capable of. Amidst his tragedy, he extricated some more living bodies and contributed his bit to save some lives. “My wife incidentally, is another poor soul, who was hit in the calamity, and whom I happened to take out from the debris that day.”
Life left him bereft, of money, job and worst of all, his dear ones.
And so fate brought him to Pune to start everything from scratch.
The small child, who embraced his first toy- a car in all rapture, who used to be all in awe of his father’s taxi, and who shocked his Dad when he learnt driving by stealthily taking the taxi on a trial spin one evening while his father was having tea after a long day work, was now going to start an entire new life and his daily bread by driving taxis.
But he refrains from cribbing or making any complains at all. Uparwallah has been very kind in his estimation. “Wo niyat dekhta hai, aur barkat deta hai.” Staunchly believes Salim.
Clinging relentlessly to Niyat and himmat thus, Salim’s hopes and dreams now envelope his son Sulaimaan. “He too loves cars like me. I want him to rise high in life and do things much beyond I could dream of. I will give him the best I can.”
My stop has come as he finishes his sentence and all I can give him back for a hard-hitting inspirational ride is one heartfelt - Ameen!