Creating an HPC application framework from a mere HPC application automatically

1. Rob Armstrong SNL Creating an HPC application framework from a mere HPC application automatically

2. The problem is that application developers see themselves at the center of the universe... ... and not framework developers as it rightly should be... • Explains the “Cactus” phenomenon: • domain-specific, special purpose framework that has a home-grown audience. • framework is adapted from a successful application in that domain. • has a “big event loop” architecture that is less flexible. • aside: why is this? • has difficulty expanding outside its original domain. • CCA has grown the ability to do this. • Onramp->Bocca->Ccaffeine->CCASpec->Babel

3. Autogen'ing an HPC framework from an existing monolithic application /* CCA->COMP(Test)->BEGIN CCA->PORT(Hello)->BEGIN */ int Hello1(int x, char *arg)‏ { int i; i=2+3; return i; } //CCA->ARG(2,out)‏ int Hello2(int x, char *arg)‏ { int i; i=2+3; return i; } • Identify components and methods with Onramp. • The hard work is still there: • how is this to be decomposed into components? • what is the execution graph for these components? • Does it make sense? • Iterate

4. Re-create the application using existing tools... • Use Appgen to create a main() that marshals the components. • Componentized framework does exactly what the original did... • but now is a plug-able framework • presumably more amenable to collaboration • is full-up CCA compliant, has a generally expandable architecture. • Resulting framework is, and can remain, an automated derivative of the original • A framework is generated as part of the build process for the original application.

5. Workflow for “automatic” framework generation... CmdLineBuilderViewForHuman::~CmdLineBuilderViewForHuman() { out = NULL; bm = NULL; } /** Implements ComponentChangedListener. Signal a change in the Component's status. */ void CmdLineBuilderViewForHuman::componentChanged(ComponentChangedEvent* evt) { if(out != 0) { fprintf(out, "revalidate \"%s\"\n", evt->getComponentInstance()); } } void CmdLineBuilderViewForHuman::setOutputStream(FILE *out_) { out = out_; } void CmdLineBuilderViewForHuman::setBuilderModel(BuilderModel *bm_) { bm = bm_; bm->addComponentChangedListener(this); } void CmdLineBuilderViewForHuman::displayPallet() { if(bm == 0) { pn("!no pallet available"); return; } std::vector< std::string > pal = bm->getPallet(); pn("Components available:"); for(int i = 0;i < (int)pal.size();i++) { pn(pal[i]); } } void CmdLineBuilderViewForHuman::displayInstantiatedComponents() { if (bm == 0) { pn("!No instantiated builder."); return; } ::std::map< ::std::string, ComponentInfo_shared > arena = bm->getArena(); boolean shown = false; ::std::map< ::std::string, ComponentInfo_shared >::iterator pos; for ( pos = arena.begin(); pos != arena.end(); pos++) { ComponentInfo_shared ci = pos->second; if (ci == 0) { pn("!corrupted arena!!"); assert(ci!=0); exit(1); } p(ci->getInstanceName()); p(" of type "); pn(ci->getClassName()); shown = true; } if (!shown) { pn("!no instantiated components available"); } } void CmdLineBuilderViewForHuman::displayComponentInfo(const char *instanceName)‏ { ::std::map< ::std::string, ComponentInfo_shared > arena = bm->getArena(); ::std::map< ::std::string, ComponentInfo_shared >::iterator pos; pos = arena.find(instanceName); if (pos == arena.end()) { pn("!displayComponentInfo component instance not found"); return; } ComponentInfo_shared ci = arena[instanceName]; if (ci == 0) { pn("!corrupted arena!!"); assert(ci!=0); exit(1); } pn("------------------------------------"); p("Instance name: "); pn(ci->getInstanceName()); p("Class name: "); pn(ci->getClassName()); pn("------------------------------------"); p("UsesPorts registered for "); pn(ci->getInstanceName()); pn(""); ::std::vector< UserPortData > pi = ci->getUsesPortRegister(); size_t i; if (pi.size() != 0) { char tmp[40]; for(i = 0; i < pi.size(); i++) { sprintf(tmp,"%d. Instance Name: ",int(i)); p(tmp); p(pi[i].getPortName()); p(" Class Name: "); pn(pi[i].getPortType()); } } else { p("!No UsesPorts in "); pn(ci->getInstanceName()); } pn("------------------------------------"); p("ProvidesPorts registered for "); pn(ci->getInstanceName()); pn(""); const ::std::vector< ProviderPortData > ppi = ci->getProvidesPorts(); if (ppi.size() != 0) { for(i = 0; i < ppi.size(); i++) { p("Instance Name: "); p(ppi[i].getPortName()); p(" Class Name: "); pn(ppi[i].getPortType()); } } else { p("!No ProvidesPorts in "); pn(ci->getInstanceName()); } pn("------------------------------------"); } void CmdLineBuilderViewForHuman::pullDownComponent(const char *className, const char *instanceName) { p(instanceName); p(" of type "); p(className); pn(" \nsuccessfully instantiated"); } void CmdLineBuilderViewForHuman::connect(const char *fromInstance, const char *providesInstance, const char *toInstance, const char *usesInstance) { p(toInstance); p("))))"); p(usesInstance); p("---->"); p(providesInstance); p("(((("); pn(fromInstance); pn("connection made successfully"); } void CmdLineBuilderViewForHuman::disconnect( const char *fromInstance, const char *providesInstance, const char *toInstance, const char *usesInstance) { p(toInstance); p("))))"); p(usesInstance); p("-\\ \\-"); p(providesInstance); p("(((("); pn(fromInstance); pn("connection broken successfully"); } provides port Component1 uses port Component2

6. Needful things ... SWIG-like typemaps a = dict()‏ a[“asdf”]=”;lkjj” a_component.a_method(a)‏ • Onramp only supports argument types that are convertible to Babel, many needed types do not exist: • Types that are not supported in all languages: e.g. hashtables. • User-defined types that need to be preserved across languages. • Nothing new here, for this purpose Babel is no different than SWIG: most of SWIG is typemaps (measured in lines of code). Arg marshaling in Py IOR in Arg marshaling in C IOR out Arg marshaling in Java /* in java AComponent */ static void a_method(Hashtable a) { String value = (String)a.get(“asdf”)‏

7. Toolkit Update • Masha has made progress with Scalapack • Standards for extra-bocca components: • https://www.cca-forum.org/wiki/tiki-index.php?page=Component+Builders+Guide+Sans+Bocca • (Just mouse down from the “Toolkit” menu)‏ • BoF Friday • Another Telecon in a week. • Possible “intensive” efforts for a small number of toolkit components at a time • you visit us • we visit you