Enterprise Applications, Information, Web
Solutions Architecture and Development
Application/Data/Object/Hub design and development
Interface/Messaging/Transformation processes
ETL/EAI/EII/Application Development
Contact
/commapp/images/emailLetter.gif
Home
Blogs
Socialize
Shop
Site Map
Admin
Exit
Latest Blog...(See all Blogs)
Data Science for Information Technology By Angelo Adduci On 2017-09-17

A Data Science Application (DSA) follows standardized steps, from initial project proposal/requirements to final project deliverables. This is the normal flow for any Data Processing application. Specific to DSA are the steps in between.

 Caretakers of an end user application initiate a project in order to produce a desired result.  This can be as simple as a Report, to the creation of a new system. For a DSA, this will most likely be the creation of datasets (files, database tables/SQL results) for use in Analysis/Reports as well as feeds to external processes.

Business Analysts utilize their knowledge, as well as attained knowledge  from  Subject Matter Experts (SME), in order to produce a Requirements document. This is a non-technical description of what is being requested for a desired result, as well as information/data sources needed for the result(s). These data sources can be anything from existing reports/online screens, to named data entities from a data dictionary/database catalog.

Vist http://www.javadatascience.com/ blog for Java Data Science Applications and community of informed professionals. 

Wikipedia

History of Data Science 

 
SQL Coding Standards By Angelo Adduci On 2017-02-25

For a successful development process, standards are needed to maximize performance and time to market, as well as future maintainability of the developed projects.

Normally, more than one standard is used to govern different aspects of development. Frameworks are one, which requires the new development objects to use the underlying Frameworks objects, thus ensuring a standard usage.

Another is for the Programming objects, even those running within a Framework, which have a standards for the actual coding statements.

These coding standards are meant to enhance readability of the coding statements, minimizing understanding time and in turn development time and coding errors.

There are always different ideas on style and what constitutes a "best" standard. However, what is important is that a standard be defined and then be adhered to.

Following are two SQL SELECT statements to illustrates standards.

Both are the same in result set requested and produced, but differ in coding style/standards.

 

(A) STANDARDS followed

-- Comments on business/application requirement for this SQL

SELECT CUST.FIRST_COL

              ,  CUST.SECOND_COL

              ,  ACCT.FIRST_COL

              ,  ACCT.SECOND_COL

   FROM CUSTOMER CUST

     JOIN ACCOUNTS ACCT

         ON CUST.FIRST_COL = ACCT.FIRST_COL

      AND CUST.SECOND_COL = ACCT.SECOND_COL

WHERE CUST.FIRST_COL = 'some value'  

      AND ACCT.SECOND_COL = 'some value'  

 

ORDER BY CUST.FIRST_COL

                    , CUST.SECOND_COL

;


(B) NOT STANDARD

SELECT A.FIRST_COL, B.FIRST_COL, TAB2.SECOND_COL

FROM CUSTOMER A, ACCOUNTS B

WHERE A.FIRST_COL = B.FIRST_COL AND A.SECOND_COL = B.SECOND_COL AND A.FIRST_COL = 'some value'  AND A.SECOND_COL = 'some value' 

ORDER BY A.FIRST_COL, A.SECOND_COL;

 

Notes

1.  One can observe, even without knowing the standards being followed, that the standard (A) is easier to read then (B)

2.  First line(s) should be comments that describe the business requirement that this SQL is performing

3.  Verb (SELECT) appears on first line of SQL statement along with first column/function of the result set.

4.  Any more result set columns/functions will follow with only one per line prefixed by a comma.

5.  FROM statement follows with first table being on same line as FROM.

6.  Table qualifiers (CUST, ACCT) need to be more descriptive than a single letter (A, B...). This greatly aids in visually associating a referenced column with it corresponding table, matching on the qualifier.  

7.  Other tables that require joining follow, with one referenced table per line starting with the JOIN (LEFT/OUTER etc...)  statement. NO COMMA for a prefix to the joined table as (B) has. Reason being we want to use the ON statement for the joined columns of the tables being joined, instead of placing them in the WHERE clause. This will separate the joining columns from other columns being used in the WHERE clause reducing confusing on columns being used for the joined tables

8.  Each additional verb that is needed (WHERE, HAVING, ORDER…) then follows with one verb and one referenced columns per line (prefixed by column)

 

 
Data Governance Control By Angelo Adduci On 2016-12-31

Data Governance Control.png

Business application systems are information systems. They consist of Data and processes (Functions) that act on that data. 

Information is the central component to a business and is used to define it. For an effective information system, controls need to be in place that will govern the definition and access (processing) of that information. 

This governance involves SME (subject matter experts) defining/approving the data definitions (metadata), as well as the business functionality.

Software (MIS) is used to create the data stores that house the meta-data, and providing access to it as information. Existing systems have the meta-data built from already used components. New systems will have this build be part of the development process.

For RDBMS, this is included within the database catalog tables.

The goal is to produce an accurate, accessible and maintainable Information system in a secure and cost effective manner. 

Data Governance is the ways and means of putting controls on the information to achieve this goal.

 
Data Governance Control By Angelo Adduci On 2016-12-31

Data Governance Control.png

Business application systems are information systems. They consist of Data and processes (Functions) that act on that data. 

Information is the central component to a business and is used to define it. For an effective information system, controls need to be in place that will govern the definition and access (processing) of that information. 

This governance involves SME (subject matter experts) defining/approving the data definitions (metadata), as well as the business functionality.

Software (MIS) is used to create the data stores that house the meta-data, and providing access to it as information. Existing systems have the meta-data built from already used components. New systems will have this build be part of the development process.

For RDBMS, this is included within the database catalog tables.

The goal is to produce an accurate, accessible and maintainable Information system in a secure and cost effective manner. 

Data Governance is the ways and means of putting controls on the information to achieve this goal.

 
Java Batch Development Framework for The Enterprise By Angelo Adduci On 2016-12-31

 

 

A batch data processing framework that simplifies, standardizes and optimizes the development and execution process.

This results in a quicker time to market process, with reductions in problems and errors associated with not utilizing a framework.

Java Batch Enterprise(JBE) is a framework that utilizes standard Java classes (POJO), forming the basis of development and execution

A Java Batch application utilizes JBE for easy to use/optimized data access to multiple Sources and Targets, with embedded HTML/Text reporting control.

 

Some benefits...

• Optimized Batch Processing of enterprise data sources and targets, including ETL and HTML    reporting.

• Reduced Cost Savings and Time To Market using industry standard Java and JDBC.

 No added software package to buy or depend on for it’s existence.

 No Need to train or hire high priced personal to maintain. Use any professional Java    developer.

• Reduced Developing and maintenance time and errors. •Reduced Redeployment cost/time of      development/executing platforms.

 In the future will you need to go from a Windows development platform to UNIX or Linux?

 Anywhere the Java JVM runs, and Java tools for development, could be utilized.

Standardization of development and execution with JBE as the standard.   

 

How is it used?

Development is performed within any Java IDE, including open source Eclipse and IDE’s built upon Eclipse.

These include MyEclipse and the family of Rational Application Developer IDE’s (Rational Developer for Z, WebSphere Application Developer )

 

Execution can be done via standard batch files *.BAT (Windows), shell scripts (UNIX/LINUX/AIX/USS) or JCL if running on z/OS. A self contained directory structure allows for the running of Java jar files (packages/program), by a simple drag/drop of a data file into a *.bat file, which contains the executing program. 

Alternatively, the *.bat file can include the data file name, with no need to drag/drop.

Execution can also be done within the Java IDE. This is an important ability for use in the development process.

 

For example, to produce an HTML report using an  SQL statement as input (or an input File record), only one new class needs to be developed.

This new class would  extend an existing java class contained in JBE. The input SQL could be contained in an extended  method of the new class, ora file that is referenced in the executing *.bat file or referenced in the Main programs properties file.

Executing the jar/package of this new class would produce the HTML report.

 

If one needs to be in control of  each row returned from the SQL result set (or input file rec),  a new method within the new class is created that overrides the behavior of the default report process.  The application would then be able to  perform any Java operation on each column/field and decide to write out a report record including changed content.

 
Software Engineering - SDLC By Angelo Adduci On 2016-12-31

For professional IT projects, all software goes through a company defined process, where documentation is created and approved before any development/changes are started.  These often include BDD (business design if software is meant for business line usage) and TDD (tech design) that constitute the Program Requirements. Individual program specifications can then be produced, adding the nonfunctional requirements. Diagram documentation, quite often using UML, is created, so as to define the flow of data and processes.  A “picture is worth a thousand words” is very evident here, and diagrams serve as discussion points for interested parties.

Individual programs quite often go through a “walk though” meeting, where the program is presented and looked at by those deemed to be SME’s (subject matter experts) so as to ensure that requirements are being met and adherence to corporate standards.

Approvals are performed by stake holders, such as DBA’s and other system folks, as well as any other downstream software that is dependant/use the results of the new/changed software. This is to done to foresee any possible negative impact that would require a resolution, and to enforce standards

For testing, Use Cases are imperative so as to ensure that all requirements are meet. These are defined by the Business Analysts in consultation with the development team, so as to include the nonfunctional requirements needed for testing.  They are also used when future changes are made, serving to ensure that past requirements are still being adhered to.  Use cases are not static. They can be added to for new functionality, or changed if past requirements were changed.

While  consulting  at FISA (NYC financial system),  I was giving the responsibility of  ensuring that Accounting Interfaces from over 12 NYC agencies,  were processed successfully by FISA’s  new accounting system (FMS),  in format and content.

  • Program requirements

Input XML files containing an Agency’s accounting interface will feed this process. These files will be validated for proper format and valid content.

Files failing validation will be rejected, while those deemed valid will be inserted to the appropriated Oracle table.  

Failed and successfully processed files will have an appropriate message written to the output HTML report.

  • Design constraints

Java is required.  The JDOM class library will be used for parsing of the XML files.  JDBC will be used for access to the Oracle database, where the accounting data resides. All reports need to be created in HTML and held on the Web Server for user access in a secure manor.

For an efficient process, database calls will only be made after the transaction passes through the appropriate validation. All SQL Selects are required to have the Primary Key index utilized.

 Log4j will be used for logging of errors.

 

  • Functional requirements.

The input XML file is first validated for proper XML syntax using the JDOM API. Badly formed XML files will be rejected with an error report written. No further processing will be performed for this file.

Each data item tag within the properly formed XML file needs to be validated against the associated Oracle table column.  The Meta Data repository of Oracle will be used to select all the columns contained within the table.  A tag name that has no matching column name will be rejected.

If the table contains a column that has no associated XML tag, then the record will also be rejected.

All files passing validation will have an SQL INSERT statement created, populated with data from the XML tags, and inserted to the Oracle table using JDBC. 

  • Nonfunctional requirements.

Badly formed XML (or empty file) will be caught as an Exception by JDOM. 

All SQL Exceptions will be caught by program, and a database ROLLBACK will be initiated.

The SQLCODE returned from the JDBC INSERT call will be interrogated for a successful completion. Any Exceptions will be written to the error report, ROLLBACK if appropriate, and stop processing.

All caught exceptions need to be reported on, detailing the error with an appropriate message, the method and line # of the  exception as well as the process flow to that exception.

If a tag is being rejected, processing will continue with the next tag, so as to have all rejects written to the report file.  However, any database Exception will cause the process to ROLLBACK and exit.

 
Application Mindset By Angelo Adduci On 2014-01-26

 Application Processing

Applications are the automated solutions to business processes. 
Information sources of paper using file cabinets for storage,  are best served as computer data stores (databases, files) and digital displays ranging from Monitors to smart phones. Although paper and physical folders are still used, they are copies of the stored data and not the source. 


Enterprise applications require more than a display and database. Complex business practices require a multitude of processing in interactive, transactional and batch modes.  
The architectures of information models, processing flows, user and external business interactions need flexibility for a cohesive and maintainable infrastructure. Equally, these architectures require a solid foundation for intensive usage and associated security within their own domains. The soundness of any system is founded in it's least sound component.  

  Information Modeling

Data is a valued asset of a company. It is not owned by any group or individual but shared by those deemed to satisfy a business need. Security requires that different user roles have different access rights to the data, on a "need to know/process" basis. 
Data on it's own is just storage. It becomes useful  when it is  Information, which is composed of data being presented and manipulated as business functions
Information that is well defined, utilized and stored in an efficient and secured manor, serves to define how a business functions and its bottom line.


Data Modeling

Defines the relationships among the many data items by grouping them as common attributes. These groups of attributes can be related to other groups by common keyed attributes. These models would then be implemented in an efficient physical model for speed in access and storage. All of this needs to be performed with the security of data as a  central focus.


Class Modeling

The means of defining the business processing requirements of its modeled data. The implemented Class model, in effect,  becomes the programming (code) of the system. The use of a Class architecture for an applications processing results in minimized development (time to market), maintenance (change of market)  and processing exceptions(delays to market).


Interface Modeling

Shared resources among diverse systems is the domain of Interface modeling. The need for a common description of the resource is essential for a workable model. XML is the industry standard language for interfaced data. XSD is the means to modeling these XML interfaces, providing the rules and validation to ensuring a workable architecture.


Information is the result of the implemented Classes interacting with the implemented (physical) data model.  All roads to a company's information needs to be through it's Classes, not its physical data model, thus insulating an application from changes to the data model (or system that houses that data model)

  Web

Presentation

The web was born with the advent of HTML. Previously, information exchange over telephone lines existed as text based messages, similar in nature to the IM messages of today.

HTML is a tag based language which allows for a developed presentation of information, images, audio and other objects. Data can also be entered, allowing for user interactions with application databases and for social network applications to function.  XML is a superset of HTML, which is a specific implementation of it.


Services

Web Services, which are transactional in behavior, allow for the exchange of data in a real time manor over a web based infrastructure, allowing for connection via a URL. 

This exchange can occur across the enterprise and as a business to business (B2B) exchange. However, efficiency comes into play and not all data should be exchanged via web services. Batch (FTP) or Message Que exchanges have their place for large scale data exchanges and exchanges not requiring real time information. It should be noted that Message Que hubs (MQ) can behave as the transport layer for web services. 

 

Web 2.0

Social Network sites have changed the web landscape. From being content and data driven, the new web isuser and community driven . Web 2.0 sites are architect to provide an infrastructure that allows for the creation of information by users about themselves and their interests, shared by the community. 
Advanced sites allow users to become part of, or even create their own communities of  like minded interests, causes and goals.  
Collaborative sites, such as Wiki , allow for any user to create or modify existing content for a specific subject matter.  

 

Business Web 2.0

Most Business web sites are nothing more than "store front signs" with pretty images, videos and graphics. They inform of services or products provided and might even display prices, but they live on a self imposed island, offering nothing more than just a "this is what we sell and here is the price". 
Websites need to be a reflection of their  business. They need to demonstrate subject matter expertise of what they are offering, and allow a user experience of being part of the community the business lives in. This community is made up of not only other users, but employees of the business, suppliers and providers to that business. 


   

  Architecture

The defined foundation of any structure or system, where all else is built upon. This can range from the home you live in,  to the behavior of an applications system and the structure of it's data stores.

One only needs to know how important a foundation is to one's home, to understand the importance of it in a computing system. 

To architect is to design first and build later, with the best architectures often built by those having been the best builders. 

 
MBO Partners By Angelo Adduci On 2014-01-26

reprinted from MBOPartners

MEET ANGELO ADDUCI

Angelo Adduci has spent more than thirty years as an independent IT professional and over that time, he’s seen the industry evolve. According to Angelo, the IT sector is much more specialized now – one reason he’s focused in on being an expert in both his field and in certain verticals like financial services. Angelo’s role in technology surrounds the business systems and databases. With a client, where he usually works on-site, his duties can include  building enterprise software systems as an architect or very hands on development, data scientist, and SOA Patterns documentation .

Having worked with clients like TD Bank, US Trust, Morgan Stanley and FISA, Angelo has decades of experience under his belt -- and has gained a wealth of knowledge. His duties have ranged from creating computer systems from manual business systems, restructuring existing business systems, to massive data and system migrations, and everything in between. By working with just one client at a time, on a designated project, he takes IT projects from start to completion, and functions as the intermediary between the business side and technology division of an organization. Understanding both sides of a business is a valuable asset that allows him to effectively communicate throughout an organization. Being an independent professional has allowed him to diversify his clients while becoming an expert within the financial industry, and has also allowed him to pursue other interests and professional endeavors, like building his own website and starting a family business.

HOW LONG DID IT TAKE YOU TO BUILD A STEADY BUSINESS? HOW DID YOU DO IT?
Early on, it was through word of mouth referrals. I started by building a good reputation at one client, and keeping in contact with various clients who referred me to other companies and agencies.

WHAT DO YOU ENJOY MOST ABOUT BEING AN INDEPENDENT PROFESSIONAL?
I enjoy working with technology. If you’re going to be independent, you have to really like what you’re doing. I enjoy focusing on technology and find different jobs with different technologies. Independence has also allowed me to pursue other professional endeavors, businesses and websites.

WHAT CHALLENGES YOU MOST IN YOUR INDEPENDENT WORK LIFE?
Getting projects can be a challenge. Working on a contract basis primarily in the financial services industry has been difficult recently due to the industry’s recent setbacks.

“If you're going to be independent,
you have to really like what you're doing.”

 

 

WOULD YOU RECOMMEND INDEPENDENT WORK TO YOUR FRIENDS?
If they intend to stay in the technical industry, and if they’re interested in taking control of their own learning, I would suggest it. If their focus is in management, it may not be a good career decision.

HOW DOES INDEPENDENCE WORK FOR YOU IN NEW JERSEY?
Being a contractor, I have the time to work on my family business, DNA Lady, www.dnalady.com.

AS AN MBO ASSOCIATE, HOW HAS YOUR EXPERIENCE BEEN?
I have always felt that I am a "partner" with them and not just another paycheck to process. They have looked out for my best interest during the client contract process, assuring that the client was not taking advantage of me, while being flexible with the client needs. Additionally, my questions are always answered on a timely basis.

WHAT TIPS DO YOU HAVE FOR OTHER INDEPENDENTS?
It’s simple. Gather the contacts you’ve made throughout your career and reach out to those people. 

 
Profile By Angelo Adduci On 2017-08-05
angelo_lrg.image

Angelo Adduci

View Angelo Adduci's profile on LinkedIn

  • Full project life cycle (SDLC) for enterprise business systems on distributed and mainframe platforms.
  • Provide architecture, application(on-line, batch) , data scientist/engineering (data analysis/mining/modelling, SQL query writing/optimization, distribution) and web (services, presentation, content) solutions at group or enterprise level on multi-tiered platforms
  • Leadership/guidance role for development team. Can be very hands-on with extensive development experience, or devote full time in architecture/leadership role.
  • Innovative and automated solutions, minimizing resource needs for more efficient, timely and cost effective projects.
  • Professional, dedicated, motivated, team player, personable with excellent verbal and written skills.

Technology Skills and Portfolio.

This Site.
Utilizes xTendSite, a PHP 5.3+ Architecture framework that I designed, developed and implemented using pure PHP Classes.
  • SMARTY Template Engine 3.1+.
  • Mark-up: JScript/XHTML/CSS/XML
  • Development IDE: Zend Studio for Eclipse
  • RDMS Database: MySQL 5.x
  • Web Server: Apache 2.2.4
  • Operating System: FreeBSD(BSD UNIX)
  • Development Server: Redhat Linux Enterprise Server 5.x using Apache and MySQL
Financial and Business Applications.
  • Securities processing
  • Mutual Funds
  • Portfolio Management
  • Custody
  • Purchase and Sales
  • Security Lending
  • Accounting
  • Payroll/AP/AR
  • Telecom/Billing
  • Customer Accounts
  • Insurance
  • Pharmaceutical
Data Engineering
  • Data Analysis/Reporting/Presentation
  • Development with Java JDBC, Stored Procedures, COBOL, Utilities, Tools
  • Access/Interface/Extraction/Transformations
  • Optimized complex SQL writing
  • Modelling for RDMS and OO Class libraries (Java/PHP).
  • Distribution design and implementation among distributed platforms.
  • XSD modelling for XML interfaced systems.
  • DB2, Oracle, SQL Server, MySQL
SOA
  • Enterprise Service Bus
  • WebServices and Application Servers
  • UDDI implementation and governance.
  • EAI, EII, ETL.
  • XSD, XML.
  • UML design diagrams.
Mainframe Tools.
  • Z/OS-MVS/UNIX(USS)
  • DB2 zOS V10
  • SQL/StoredProcedures
  • IBM DB2 Data Studio
  • COBOL-LE
  • JCL
  • CICS/MFS/SDF
  • IMS DB/DC/BMS, VSAM
  • Rational Developer for Z
  • ISPF, TSO, JES
  • Endevor
  • Websphere MQ
  • Websphere AS
  • JAVA
  • USS Shell
  • REXX
  • Gentran GIS
Management Tools.
  • Agile/Scrum Development Methods
  • UML, TDD
  • PowerPoint, Visio
  • HP Quality Center
  • Word, Excel, Outlook, Project, Share point