Big Data Analytics scopes for your context!

Lets first try to point your context. Your context can be near these below situations:

Below are descriptions of above table:

1. You have a transactional system ( e.g. Billing or POS ) and it is using a RDBMS has its own reporting.

2. You have several transactional systems not integrated and they are using same/different sort of RDBMS each have their own reporting.

3. You have several transactional systems/ERP/CRM more or less integrated with each other or using a middleware server but still each have their own reporting

4. You have several transactional systems/ERP/CRM integrated with each other or using a middleware and you do not have separate reporting server

5. You have several transactional systems/ERP/CRM integrated with each other or using a middleware and you have a separate data warehouse with BI reporting server

Lets try to think what can be the scope you can think for your context extend toward big data analytic:

1. Transactional Systems to HDFS ( Hadoop Distributed File System) to RDBMS data warehouse to OLAP based analytic server

2. Transactional Systems to RDBMS data warehouse to HDFS data mart to hive/impala to olap based analytic server

3. Transactional Systems to HDFS to hive/impala OLAP based analytical server

4. Transactional Systems to HDFS to big data api based analytical server

Lets try to match possible better match in context and scope of the journey.

if you are in context 5: 
You are dealing a mature enterprise, already you have ETL (Extract, Transform then load) in place, here you can introduce Big data for ELT ( Extract, Load then transform), it will make transformation life easier as all data is in hdfs. Which is basically above scope 1. If your BI tool has big data plugin for hive or impala, you can adopt HDFS also there and impose OLAP layer on that for reporting so existing reports may work same. 

If you are in context 2,3 or 4 then scope 3 sounds really a better option to me for you.

And in case of context 1 I think simple is to go with scope 4. 

There can be several other combinations and possibilities may suite you enterprise better than I have portrait here. But I think it will give a clue to think before moving to Big data analytics adoption.

Mahmudur Rahman Manna

A dedicated software engineer in java platform. Acting as Lead Software Architect internationally.

Founder of Mannaco Microsystems and Co-Founder of O' Source Bangladesh Ltd. , EUSIA BD Ltd. and Newgen Technology Limited.
Primary contributor to start Liferay development practice in:
Codemate and Zensar Technologies Limited.
Mentor and Hands on trainer of Liferay, Apache Ofbiz and Pentaho BI.
Professional Consultant of Agile practice establishment and build & release management automation.

Showing anger in constructive manner is good for career

There are always complaints about my anger. I understand people take time to face me only for this reason. And a bit clever guys take facility of this by creating situations where I loose my temper and leave the control to them. Now its not something I can remove from my nature completely. I read many articles and discussed with many people. And found suggestions like think before you talk, take a break. But end of the day when its rush and many mistakes around, I again the same person. So I stopped acting that I don't get angry rather started to think how can I use it in constructive manner. This article I found more logical - Showing anger is good for career.

After studying several I found 7 points which is a bit working for me as I feel it touches the root thoughts of mine and helps me use my anger in constructive manner.

I do not have permission to rant a situation

Its my house or my workplace, normally when I am superior, a responsibility works inside me which gives a authority to guide or correct others. Every day there is possibility of new situations/problem may arise which I have also not seen earlier and have to lead to solve that. And initially all comes to participate to solve that willingly but this the anger which creates a harsh situation and spoils the freshness and rant the situation. 

Who gave me the authority to rant the situation?

Shouting will create more noise

Normally anger arises from noises that surrounding creates. And we think a big shout may bring silence but normally it not. It rather brings more noise and dissatisfaction in others which bursts slowly and lasts longer. 

Frustration is way to innovation

One of the main reason of anger is frustration. Frustration normally comes when I am failing but every time I fail, I cannot stop my life but I always have to give a new try with new idea and honestly it normally far better than earlier. So now I think less of failure which creates anger and try to make me happy of thinking something new is coming.

Argument cannot establish right

I cannot tolerate if someone is saying something wrong in front of me. And I try to argue until he agrees on logical points. And that is the big mistake I do. For that moment for strong logics he may get silent or say some stupid things but  basically he has not agreed. But in this process I get angry. As this argument is not bringing any good then why to waste time on that rather better approach is to advise him, if he understands better if not let it for him.

I cannot assert my respect forcefully

If I force my children for something, for that moment they may fear me but absolutely it will not bring respect in their mind for me. So forcing which creates anger inside is useless if its to achieve respect.

If I want people listen me, I must listen them

Giving less importance while others are speaking will also make him give less importance while you speak, which is natural to create anger vice/versa.  

No one will understand if they are hurt 

showing anger at someone and later going back to him to say sorry wont make him understand. He is hurt, he will now show sympathy to you but not understand your point anyway. 

Image courtesy of imagerymajestic at FreeDigitalPhotos.net 

Behavioural Science – Part 2

Global Sourcing IT Industry

Mahmudur Rahman Manna ^[ref]

(if you have not read the part 1)

Team Formation and current practices:

Gaming World: Multi Agent System (Markov Decision Process)

In game theory, a discussed topic is Multi Agent Systems along with Markov decision process. Agent is an autonomous entity that acts in the world, interacting with its environment and with other agents. It can be a human, a robot, a software process etc. One of the key ideas behind this approach is that several different agents can cooperate to achieve certain goals. This requires the design of efficient collaboration protocols, of which team formation is a typical example.

Cooperative behaviour, which is one of the greatest advantages of agent based computing, has been studied from many different angles over the years. Coalitional games have traditionally been analysed from a game-theoretic perspective, but in recent years have attracted a lot of attention from researchers in artificial intelligence, especially in cooperative task completion. Several approaches for team formation and collaborative task solving have been considered including team formation under uncertainty using simple heuristic rules, reinforcement learning techniques and methods using distributed graph algorithms. To reason formally about cooperative games, several logics (e.g., Alternating Time Logic, Coalitional Game Logic, Strategy Logic) and other formalisms (e.g., Cooperative Boolean Games) have been introduced and used to analyse coalitional behaviour

This theory introduces definitions of:

-          Agent organisations,
-          Tasks,
-          Teams,
-          Rewards (to measure the performance of both individual agents and agent teams).

Furthermore this theory has now several developed algorithms and models for verifications. Off course those are to improve how we can derive more efficient and effective, near perfect results/successes.

Relevance of introducing this topic is clearer for the similarities it has with our multi provider and vendor oriented developments. In our case it’s not system or robot rather it’s us. A consciousness driven being whose self-esteem is highly counted to him/her rather than any team success and organisation gain if not it merges or matches with his/her self-esteem.  

Its better we are already dealing with intelligent and social being, who usually loves to be in team. But also we know about lots of wars the world has seen. So the dynamics of team is really complex part to keep in a steady mode and surely if there is no such mentor body to take care of that.

But the interesting part of Markov Decision Process (MDP) is its ‘reward’ part that really counts also in human cases.

MDP has four key things:

1.       State
2.       Action
3.       Probability of an  action to be in a state at a certain time
4.       Reward

And reward comes when an action achieves a certain state. 

                                                                   

                                                                 Figure 6: MDP (‘S’ refers State and ‘a’ refers action)

There is much implication of MDP in our multi provider based project development. Our try will be to relate team dynamics and MDP core concepts to bring on a table and analyse how it can help in having a better environment for individuals who are dwelling in heterogeneous provider based platform.   

References

John B. Miner, ORGANIZATIONAL BEHAVIOR Foundations, Theories, and Analyses, OXFORD UNIVERSITY PRESS, 2002

Stephen P. Robbins, ORGANIZATIONAL BEHAVIOR, Prentice Hall International, Inc.

Fonstad, Nils, and Robertson, Engaging for Change: An Overview of the IT Engagement Model, CISR Research Briefing, Sloan School of Management, Massachusetts Institute of Technology (MIT), March 2005.

 Craig Schiff, Engagement Management is Key to Implementation Success, http://www.information-management.com/issues/20040401/1000840-1.html , November, 6^th 2011.

Dr. Z , Business of Stress: Rise of the Type A Machines, http://www.stresshacker.com/2010/01/business-of-stress-rise-of-the-type-a-machines/, November, 6^th 2011.

Donald Bell, UML basics: The component diagram, http://www.ibm.com/developerworks/rational/library/dec04/bell/, November, 6^th 2011.

Ilan Oshri and Julia Kotlarsky, Special Issue on Global Sourcing: IT Services, Knowledge and Social Capital, http://www.palgrave-journals.com/jit/journal/v23/n1/full/2000129a.html#bib7 , November, 6^th 2011.

Taolue Chen, Marta Kwiatkowska, David Parker, and Aistis Simaitis, Verifying Team Formation Protocol with Probabilistic Model Checking, Department of Computer Science, University of Oxford, July 2011.

Frank Dignum, Dialogue in team formation: a formal approach, Faculty of Mathematics and Computing Science, Technical University Eindhoven.

Jos´e M Vidal, Fundamentals of Multiagent Systems with NetLogo Examples, University of South Carolina, March 2009.

S. Abdallah and V.R. Lesser. Organization-based cooperative coalition formation. In IAT, pp. 162–168. IEEE, 2004.

Roderick van cann, slinger jansen and sjaak brinkkemper, Team Composition in Distributed software development, universiteit Utrecht

Behavioural Science – Part 1

Global Sourcing IT Industry

Mahmudur Rahman Manna ^[ref]

 

This writing is to identify specific components involved in global sourcing for IT services and signify there behavioural patterns and relationships among them. Try will be to bring out key elements that drives the process to success or turns into frustration.

Introduction

Psychology, sociology, social psychology, anthropology, and political science; current organisational behavioural science discusses about these topics. When two organisations get involved in a set of processes then the significant importance of organisation behavioural science becomes stronger.  Engagement chemistry has a bold impact on and in the involved organisations.

 

Type mismatch is a usual error we have to take care of in our programming life. And in polymorphic and dynamic context, casting is a solution mostly we use. If an Interface (template or set of standard definitions) is implemented in our local context (inner organisation) and if the Interface is available in remote context then there is good possibility that our implementation will be recognisable over there. Other than that we have to customize our implementation or set our Interface there newly. Now commonly everyone is used to with the word “lookup”. In solution development normally we use this lookup with our Interface name in remote contexts to check if there is any implementation available for such interface.  

Sourcing triggers with this lookup task.  Hundreds of organisations having this lookup every day to get their suitable providers. Some of them have some standard of doing so and many of them don’t know even how to do this. The consideration of behavioural science should pop up here before starting this lookup. 

Perspectives of Different Levels

There are specific levels in each organisation.  Each level most of the time act as an independent entity with its own attitudes and attributes.  On the same issue, a level would have different messages or thoughts differ from other levels.

 

In an organisation normal chemistry happens among:

Global Manager -- Department Manager -- Team Manager -- Team member 

As they represents their level with more or less understanding of his/her environment. But when it comes to an engagement of two or more different organisations for a specific goal then the ancient chemistry not works anymore.  The cross communicative functionalities gets a rapid increase on Team Level and as it is the operational level,  constitutional wrong mapping on that level would create more trouble and cost for both companies. Our case is IT engagements, so we will focus more on IT issues.  

 

A Scenario

The sourcing may be coming out from client IT department budget but other departments get involved into it more closely. Even it gets more competitive when there are different providers and vendors. Even there are cases where in same solution different providers working together -

 

Isn’t it already a chaos, stressful to think the level of interaction required here? It rarely found as homogenous but most of the time in heterogeneous form.  And it gets more complex when a Mediator Organisation walks in and takes ownership and deals with the providers.  May be a relief for sourcing organisation managers, but the social environment that builds up here is off course not hygienic, if there is no ownership to take care of the hygiene and look into the behavioural chemistry to provide stability.

Current market packages with different terms available for IT sourcing:

-          Consultancy/ professional services
-          Product based
-          Fixed
-          Development centre/ offsite
-          Onsite
-          Hybrid (onsite offsite )
-          Virtual Captive model
-          ……

 

Team and Individual

Our focus part will be the team and individuals in team and the elements they deal with.

As we are here for responding to globalization, behavioural science can meet the challenges that help to overcome stress and absenteeism here. We will try to figure out, how these challenges can be meet by behavioural science.

-           Workforce Diversity
-          Improving Quality and Productivity
-          Improving Skill
-          Empowering
-          Coping with frequent changes
-          Stimulating Innovation
-          Improving Ethical Behaviour

      

Typical Scenario of Team formation

Off course it varies based on which engagement model we are following but typically a project initiates the whole formation.

• Projects are the initiators of engagements. 
• Projects are the initiators of changes in engagements.
• Projects come with a budget and target and a deadline.
• Projects not always come with plan.  

  

And whenever it’s there, the owner starts the “lookup” with his/her budget.  Owner wants to earn the credit of making its success within budget, if possible save some from budget.  And this parameter in lookup task filters the existing or new providers.

 

References

John B. Miner, ORGANIZATIONAL BEHAVIOR Foundations, Theories, and Analyses, OXFORD UNIVERSITY PRESS, 2002

Stephen P. Robbins, ORGANIZATIONAL BEHAVIOR, Prentice Hall International, Inc.

Fonstad, Nils, and Robertson, Engaging for Change: An Overview of the IT Engagement Model, CISR Research Briefing, Sloan School of Management, Massachusetts Institute of Technology (MIT), March 2005.

 Craig Schiff, Engagement Management is Key to Implementation Success, http://www.information-management.com/issues/20040401/1000840-1.html , November, 6^th 2011.

Dr. Z , Business of Stress: Rise of the Type A Machines, http://www.stresshacker.com/2010/01/business-of-stress-rise-of-the-type-a-machines/, November, 6^th 2011.

Donald Bell, UML basics: The component diagram, http://www.ibm.com/developerworks/rational/library/dec04/bell/, November, 6^th 2011.

Ilan Oshri and Julia Kotlarsky, Special Issue on Global Sourcing: IT Services, Knowledge and Social Capital, http://www.palgrave-journals.com/jit/journal/v23/n1/full/2000129a.html#bib7 , November, 6^th 2011.

on release date of iphone 4s - apple.com was inaccessible

process n practice

automatically we are always in a process. look into your morning .. from your coffee to first toilet :).  my wife can easily define me :).  i know you are laughing and surprisingly on yourself .. yeah mee too now laughing on myself. 

now think suddenly you found, the way you go everyday to your office that is having some reparing trouble so you have to make the journey in a different way and you have no experience of other roads (n you dont have gps on your car :D). i know you are angry on me why i am even thinnking of such thing.

 see!, how irritating is this! .. i mean having break in your normal process of  journey to your office.  but ok.. at last you reached your office. then the other day .. you know two ways to reach office. and question is now to all - 

q.which way should one go?

ans: whichever is comfortable, short, easy, less costly (this is the problem if you ask any one they may if possible ask for everything :D)

 q. but if that yesterday incident not happened then?

ans: i know myself, i would hardly try to be another Columbus. (if i am not that much bound)

i think you are also pretty much same but can you feel -'how irritating the change in process is before you know it is really good or bad'

why i titled it as process n practice - if you cannot show people directly the good taste of the change in their process better to give a pause; and here comes the practice ( in granual this is small piece of activity highly connected with time (when? or a context/situation/circumstances- i call it time identifier .. the way you define a time in history books)) . e.g. you have to write n you don't carry a pen and notebook but you have a mobile and  :) now a days smart mobiles .. you will write there (   i am doing so.. less possibility of losing :D).  see how practice grows. yeah from need and need has some connectivity with time....... - 

[will continue this writing hopefully...:D but i think it was good start to introduce the thought .. now you think :D ]

 

how i learned data warehousing and OLAP

It was my view in my academic life that only learning building a SaaS is not going to give me much scope in market as already thousands of Saas is roaming in market even as open source. And I felt like that the way corporate are adopting Saas in a adhoc basis they will be surely in trouble with having intelligent reports and it would be hard for them to create decisions from them. And I was searching such solutions which can integrate data from sporadic Saas and provide a combine view. And I found Data warehousing. And as my third years project I proposed to my supervisor the subject “Designing Data warehouse” and he wondered but accepted. But I was in sea and not getting a way to see any practical implementation my nearby, meanwhile I have covered Bill Inmon and Ralph Kimball. But it was not satisfying my thirst as all was just theory. I was dying to implement a demo of my own. And this thrust turned me to Oracle. At that time I heard that Oracle 9i got release with Data warehousing concepts. I jumped for it and installed in my PC. But still I was in dark .. not getting where to start. 

Thousands and Thousands thanks to oracle team who has included SH (sales history) schema in oracle. After getting this schema I saw a light of having my dream as true. I got bulk of data provided by them.

I have created a dummy sales transactional database and used SH schema data in a modified way to pour that. Huh! Now I have a huge transactional database and I have already an empty start Schema (SH) database.  Here comes the need of ETL now. I was so much pleased to reach in such position. Baby now it is time to do some ETL .. but how?  That time the tool “Sql loader” showed me the easiest way to do that. And created some sql scripts and succeeded to perform ETL and schedule that. I was so happy to have the data in my SH schema (Data warehouse :D ).  And I cannot forget learning building the Time table for the first time. Such a table was most interesting thing for me. Now data is there, fact table there.. time table there but how to create report from them and how to provide decision supporting reports.  And a simple query takes huge time. 

Here comes the Advanced query rollup… with clause .. thanks oracle for these.  But how to minimize the query time… :D here I learned the materialized View.. I was highly excited to implement them one by one and getting the outputs. But the reports were not satisfying me.  What is the benefit of having this much data and only some ordinary reports.

Where comes the OLAP. Thanks again to oracle. I created first OLAP metadata with PL/SQL definitions. It was really much interesting to create dimensions and form a cube. Earlier I have read lot of cube but writing one’s definition in my own hand was really excellent. WOW..  It was now sounding to me I have now a real useful data warehouse build by me. But I was failing to make my friends and teachers understand this thing as they were just trying to see some outputs. And I was getting frustrated that I was unable make them understand the benefit of data warehouse with OLAP only because of having lack of some beautiful UI.

And where comes the Oracle Discoverer. WOW that this tool was really a striker for me. I used that over my SH data warehouse and showed my interested friends and teachers and now they got what I was saying in the name of Data warehouse.

I was trying to switch my java programming job and get a data warehouse related job but interestingly found that corporate are expecting some white haired aged guy for their data warehouses and getting me in their interview boards they are not satisfied. I tried several places but I saw doubt in their eyes and at the end I did not get the jobs.

Here comes the Pentaho. An American company was searching some java guy and in requirement they have written – having data warehouse experience is advantage. WOW it sounded like it is for me. I submitted C.V. and they called for interview and interestingly I got the job next day.  And I discovered it is Pentaho with which I have to build data warehouse on Telecom billing and do reporting.

WOW its excellent . I was amazed to get “Kettle”  instead of “sql loader” for the ETL .. it is really a dynamic tool and very much useful.  Doing star schema in Mysql was also a interesting thing for me. And then Mondrian OLAP server  instead of Oracle OLAP.. I was just happy to get lost with that and then MDX with Jpivot instead of Oracle Discoverer.. I was really happy in working as it was all in Java and I was playing with it shaping all in any way I liked.

I was writing the whole to store some memory in written and also in a hope that some new  data warehouse  interested guy may read it and find some tips to go ahead.