Thursday, November 13, 2014

Investigating OutOfMemoryError

The java.lang.OutOfMemoryError error is thrown when there is insufficient space to allocate an object in the Java heap (or a particular area of the heap). That is, the garbage collection cannot make any further space available for a new object and the heap cannot be expanded further. An OutOfMemoryError does not imply a memory leak – the issue can be as simple as a configuration issue where the specified heap size (or the default size if not specified) is insufficient for the application.

A good reference: http://help.sap.com/saphelp_nwesrce/helpdata/en/97/d0de42c02c420da92a7cec346f9168/content.htm

Make sure what type of OutOfMemoryError it is:
  • Exception in thread “main” java.lang.OutOfMemoryError: Java heap space
  • Exception in thread “main” java.lang.OutOfMemoryError: PermGen space
  • Exception in thread “main” java.lang.OutOfMemoryError: Requested array size exceeds VM limit
  • Exception in thread “main” java.lang.OutOfMemoryError: request bytes for . Out of swap space?

Other useful resources:
  •  http://stackoverflow.com/questions/8966017/tomcat-outofmemoryerror-java-heap-space-24gb-system
  • http://www.javaworld.com/article/2071737/core-java/plug-memory-leaks-in-enterprise-java-applications.html
  • http://help.sap.com/saphelp_nwesrce/helpdata/en/97/d0de42c02c420da92a7cec346f9168/content.htm
  •  http://www.techpaste.com/2012/07/29/steps-debugdiagnose-memory-memory-leaks-jvm/
  • http://java.dzone.com/news/how-fix-memory-leaks-java

Sunday, May 11, 2014

Postgres CPU benchmarking on my Dell Latitude E6520 laptop

I am busy reading PostgreSQL 9.0 High Performance by Smith, Gregory


Posgres db: connected via USB2 to the Postgres db on a western digital My Book Studio Edition 2 in RAID 1 (?)



psql (9.2.4)
Type "help" for help.

postgres=# \timing
Timing is on.
postgres=# select sum(ge)

postgres=# select sum(generate_series) from generate_series(1,1000000);
     sum    
--------------
 500000500000
(1 row)

Time: 293.502 ms
postgres=# \timing
Timing is off.
postgres=# \timing
Timing is on.
postgres=# create table test (id INTEGER PRIMARY KEY);
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "test_pkey" for table "test"
CREATE TABLE
Time: 532.280 ms
postgres=# insert into test values (generate series(1,1000000));
ERROR:  syntax error at or near "series"
LINE 1: insert into test values (generate series(1,1000000));
                                          ^
Time: 0.254 ms
postgres=# explain analyze select count(*) from test;
                                               QUERY PLAN                                             
--------------------------------------------------------------------------------------------------------
 Aggregate  (cost=40.00..40.01 rows=1 width=0) (actual time=0.003..0.003 rows=1 loops=1)
   ->  Seq Scan on test  (cost=0.00..34.00 rows=2400 width=0) (actual time=0.001..0.001 rows=0 loops=1)
 Total runtime: 0.027 ms
(3 rows)

Time: 0.618 ms
postgres=#

postgres=# insert into test values (generate_series(1,1000000));
INSERT 0 1000000
Time: 6766.835 ms
postgres=# explain analyze select count(*) from test;
                                                     QUERY PLAN                                                    
---------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=17700.00..17700.01 rows=1 width=0) (actual time=159.381..159.381 rows=1 loops=1)
   ->  Seq Scan on test  (cost=0.00..15045.00 rows=1062000 width=0) (actual time=0.015..99.903 rows=1000000 loops=1)
 Total runtime: 159.407 ms
(3 rows)

Time: 159.823 ms
postgres=#

Checking CPU and memory performance before Database benchmarking

I am busy reading PostgreSQL 9.0 High Performance by Smith, Gregory


Running STREAM  (The Java version) for one CPU:
http://www.cs.virginia.edu/stream/FTP/Contrib/Java/


Average cpu bandwidth:  Copy: 16334MB/sec/cpu Scale: 13104MB/sec/cpu Add: 16253MB/sec/cpu Triad: 15534MB/sec/cpu
Total system bandwidth: Copy: 16334MB/sec  Scale: 13104MB/sec  Add: 16253MB/sec  Triad: 15534MB/sec  

 Running Greg's app: reading https://github.com/gregs1104/stream-scaling


 WARNING -- The above is only a rough guideline.
For best results, please be sure you know the
precision of your system timer.
-------------------------------------------------------------
Function      Rate (MB/s)   Avg time     Min time     Max time
Copy:       10815.5366       0.0356       0.0354       0.0358
Scale:      10758.2016       0.0358       0.0356       0.0366
Add:        11943.3337       0.0483       0.0481       0.0489
Triad:      11913.1766       0.0484       0.0482       0.0488
-------------------------------------------------------------
Solution Validates
-------------------------------------------------------------

Number of Threads requested = 2
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      13490.2814       0.0429       0.0426       0.0435

Number of Threads requested = 3
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      13238.9345       0.0441       0.0434       0.0454

Number of Threads requested = 4
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      13217.3784       0.0444       0.0435       0.0458

Number of Threads requested = 5
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      13112.7815       0.0450       0.0438       0.0471

Number of Threads requested = 6
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      12876.9001       0.0451       0.0446       0.0456

Number of Threads requested = 7
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      12968.6780       0.0454       0.0443       0.0477

Number of Threads requested = 8
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      12906.0583       0.0458       0.0445       0.0479

But running it I saw that my CPU frequency was set to 800

=== CPU Core Summary ===
processor       : 7
model name      : Intel(R) Core(TM) i7-2720QM CPU @ 2.20GHz
cpu MHz         : 800.000
siblings        : 8

Reading these links
Seems like me Linux Mageia 3 is using cpupower

 cat /sys/devices/system/cpu/cpu0/cpufreq/bios_limit
2201000   
                    


Also ran  wmcpufreq which shows your current speed and profile

watch grep \"cpu MHz\" /proc/cpuinfo

i7z_GUI

I changed to  performance

Edited  /etc/sysconfig/cpupower

 === CPU Core Summary ===
processor       : 7
model name      : Intel(R) Core(TM) i7-2720QM CPU @ 2.20GHz
cpu MHz         : 2200.000
siblings        : 8


Full trace
./stream-scaling
=== CPU cache information ===
CPU /sys/devices/system/cpu/cpu0 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu0 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu0 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu0 Level 3 Cache: 6144K (Unified)
CPU /sys/devices/system/cpu/cpu1 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu1 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu1 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu1 Level 3 Cache: 6144K (Unified)
CPU /sys/devices/system/cpu/cpu2 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu2 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu2 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu2 Level 3 Cache: 6144K (Unified)
CPU /sys/devices/system/cpu/cpu3 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu3 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu3 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu3 Level 3 Cache: 6144K (Unified)
CPU /sys/devices/system/cpu/cpu4 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu4 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu4 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu4 Level 3 Cache: 6144K (Unified)
CPU /sys/devices/system/cpu/cpu5 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu5 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu5 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu5 Level 3 Cache: 6144K (Unified)
CPU /sys/devices/system/cpu/cpu6 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu6 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu6 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu6 Level 3 Cache: 6144K (Unified)
CPU /sys/devices/system/cpu/cpu7 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu7 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu7 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu7 Level 3 Cache: 6144K (Unified)
Total CPU system cache: 52690944 bytes
Suggested minimum array elements needed: 23950429
Array elements used: 23950429

=== CPU Core Summary ===
processor       : 7
model name      : Intel(R) Core(TM) i7-2720QM CPU @ 2.20GHz
cpu MHz         : 2201.000
siblings        : 8

=== Check and build stream ===

=== Testing up to 8 cores ===

-------------------------------------------------------------
STREAM version $Revision: 5.9 $
-------------------------------------------------------------
This system uses 8 bytes per DOUBLE PRECISION word.
-------------------------------------------------------------
Array size = 23950429, Offset = 0
Total memory required = 548.2 MB.
Each test is run 10 times, but only
the *best* time for each is used.
-------------------------------------------------------------
Number of Threads requested = 1
-------------------------------------------------------------
Printing one line per active thread....
-------------------------------------------------------------
Your clock granularity/precision appears to be 1 microseconds.
Each test below will take on the order of 24587 microseconds.
   (= 24587 clock ticks)
Increase the size of the arrays if this shows that
you are not getting at least 20 clock ticks per test.
-------------------------------------------------------------
WARNING -- The above is only a rough guideline.
For best results, please be sure you know the
precision of your system timer.
-------------------------------------------------------------
Function      Rate (MB/s)   Avg time     Min time     Max time
Copy:       11542.3665       0.0335       0.0332       0.0343
Scale:      11572.0339       0.0340       0.0331       0.0368
Add:        13203.7697       0.0440       0.0435       0.0449
Triad:      13192.7874       0.0451       0.0436       0.0484
-------------------------------------------------------------
Solution Validates
-------------------------------------------------------------

Number of Threads requested = 2
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      13749.1610       0.0422       0.0418       0.0431

Number of Threads requested = 3
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      13538.3089       0.0434       0.0425       0.0455

Number of Threads requested = 4
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      13330.8034       0.0436       0.0431       0.0445

Number of Threads requested = 5
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      13221.0025       0.0446       0.0435       0.0472

Number of Threads requested = 6
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      13234.7194       0.0441       0.0434       0.0450

Number of Threads requested = 7
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      13181.8954       0.0442       0.0436       0.0460

Number of Threads requested = 8
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      13202.8297       0.0447       0.0435       0.0466

[pvz@localhost stream-scaling-master]$



Wednesday, April 24, 2013

Reading Extreme Programming Refactored: The Case Against XP post


Reading http://www.softwarereality.com/lifecycle/xp/case_against_xp.jsp

Disclaimer: I don't think I know enough about XP and Agile to call myself a expert. I havent read enough books on the subject.

But I have worked in companies where we tried XP and tried to be Agile and tried to use Scrum.

I think you need the know the Pros and Cons of a methodology or method so lets see what this article is saying about XP.


"However, they refused to allow the team to perform any refactoring on the morass of badly formed code that emerged, because they felt that now the code was written, spending more time on it would be a waste of money.

The programmers also finished-off each code module too quickly because they hated to pair program. The result was a monolithic city of heavily interwoven code with huge amounts of duplication, coupled with unit tests that took literally hours to run and left the database in an "unclean" state (i.e. needed to be picked apart manually after running the tests)."

I have seen a project before that also had the issue where the project wasn't re-factored on a regular basis and the code was very intertwined and the tests did take hours.....so could this be because of trying out XP on this project or is this just a observation that could be applied to all bad projects?

"Constant refactoring could potentially introduce lots of bugs, as dependent code is broken when you make small insidious changes to your design; but it's considered safe because of extensive unit testing."


Experienced this as well because some of the code that was done by a team under pressure didn't have enough tests and the refactoring broke a lot of functionality? Could this also be a problem related to not merging enough? Not enough code reviews? Not enough communication between teams?

"Because XP as a process does not involve getting a design spec reviewed by senior engineers, and instead places the emphasis on a constantly evolving design, this can be a high risk. "
I agree.

http://www.softwarereality.com/lifecycle/xp/xp_audience.jsp

"XP is aimed at customers who don't know what they want."
Is this really true?

My experience is that there is a business case document stating at a high level what they want. Then maybe a document with a few user stories or scenarios. Plus then the question becomes of what can we deliver maybe in 2 or 4 weeks for example. Can we deliver small bits that they can use and test?
Is this Agile?

The other thing I have seen is that the deadline has been set without grooming and planning sessions. Sometimes without even senior developers input........and I have never seen us deliver on that deadlines as it was never realistic to start with.

"There are processes that are much better suited to handling change halfway through a project - providing requirements traceability and impact analysis "
Like?

"(including support for quickly providing cost estimates so that the customer can make an informed decision about whether to authorise any changes)."
This is the one thing I haven't seen in our projects. Adjusting costs when change happens........

"For example, the customer might simply be unsure about exactly what their business requires. This is where you come in, as a business consultant: you should be presenting the customer with use cases, mock screenshots, prototypes (using quickly knocked-together, throwaway code). This all helps the customer to visualise what they really want, helping them to pin down the requirements before a single line of production code is written."

RAD?

"all the really hard thinking would have been done up front, and all the hard questions would have been asked (and answered). The problem domains have been identified; the nastiest problems carefully thought through and resolved."

I though that the we keep missing the real problems and only when we start coding do we find them and that is why XP and Agile wants to start coding quickly and show the client their understanding at the moment so that the client can also see if this is really what he meant and also find the real hidden technical difficulties early rather than later?

" In other words, the developers get very excited about an idea they (or the in-house customer) have had, and in chatting about it they become even more excited. In it goes. But the real customer (that reticent, cigar-chewing man back at Customer Towers) might not be quite so appreciative of this shiny new feature."

Mmm this is also true..........I have seen teams working very hard creating code and having a client that keeps them busy but did they achieve what the "board" or the big boss wanted overall? Could the client representative also have lost his view during the projects and have gone off in a tangent? Did we have a person that upfront challenged the client to clearly think about what he/she wanted?

Stupid question...........so you create stories, you add them to for example Jira. Devs go and do the work and write the tests. Is there an overall view of all the Stories to explain the functionality in the application that you can give to a CEO?

I sometimes feel there is a massive list of tasks over 6 months and then how many off them was removed in later sprints for example when the client realised that a story is incorrect or irrelevant? How do you extract the final stories that have been implemented in your application? I am sure the experts can give me the answer :)

http://www.softwarereality.com/lifecycle/xp/four_values.jsp

" And so, through countless waves of throwaway code, the design gradually (hopefully) starts to evolve. To counter this time-consuming, high risk approach, we are encouraged to keep everything simple."

What is simple? I have seen teams almost go into a constant cycle of refactoring to find the perfect solution to get to the perfect OO design. Isn't this costing money? Is the team still moving forward? Is this still simple? What is simple? What is perfect? When will we stop?

One of my friends send me this link which is very interesting on the topic of Easy and Simple......I don't agree with everything and it is so easy to misunderstand this presentation (there I go being vague and not providing proof...) but still it makes you think:

http://www.infoq.com/presentations/Simple-Made-Easy by Rich Hickey

" A common example is to produce a "framework" due to some unknown, hypothetical future requirement. If the requirement isn't there now, don't code for it."

Agree. Plus why write a framework? Can you write a better persistence framework? Or web framework and should you be doing it at your clients expense? Ok going off topic.....
"The result (as described above) is that in XP you end up writing code which you know is going to be rewritten later."

Is this really true? Aren't wegoing to expand on the code of yesterday? I don't think we are going to rewrite it.

"The difference with XP is that customer negotiation is arranged so that it runs throughout the project, and is fundamental to the way the process works at every stage. Due to the nature of XP's contracts, contract negotiation effectively takes place every time a programmer talks to the on-site customer.
I would suggest that your programmers are probably not the best people to handle the finer points of customer negotiations"

I agree

"Then the actual problem hit me. It's actually a general problem with XP, and its "throwaway" approach to production code. The programmers are encouraged to write with an attitude of "I'll just start coding and see what happens." If they end up writing garbage, it doesn't matter because they can just throw it away, go home clean, and start again tomorrow. Same goes for the design. Let it evolve, and see where it goes. The collective consciousness of the XP team will nudge the so-called design to some place or other. When it's there, if it's wrong then we'll just collectively nudge it somewhere else."

Isn't this where senior devs will come into play? Or shouldn't it become clear in daily stand-ups that we are not moving forward and that something is wrong? But again this creates an effect that teams are busy but they could be rewriting the system everyday to get to that "perfect code" but then the Simple value should kick in here and stop the developer?

"You just need to be good at what you do, as with any skilled vocation. You need some aptitude and imagination. You don't have to create something to know whether it is going to work or not. Just think ahead, factor in as much as you can without losing sight of your goal."

I agree with the thinking ahead part and that skill and experience is needed.

"Despite its flaws, XP's practices can also be tweaked to make the overall process more robust. Don't get too caught up in blind adherence to one methodology. Try not to end up chasing your own tail in order to be agile. Software agility is useful, but it's simply a means to an end."

I agree. Learn different methodologies but  don't follow them blindly and be aware of the pitfalls of any method or methodology.

"Apply whatever practices that work for your project. Read the Steve McConnell books cover to cover (he's got it right, including the Evolutionary Delivery stuff). Use what's right for you."
I agree even more with this statement :)

"RUP" I also like to try this again..........plus I think RUP has adapted in the last few years. Anyone with experience of using RUP?


""Extreme Programming Refactored

In this book (co-authored with Doug Rosenberg), we suggest some modifications to the XP practices (plus some additional practices) to achieve XP's agile goals in a more rigorous and less risk-laden way."
Would be interesting to see the modifications........

Overall I wonder how this article would change after reading Kent Beck's second version of his book?







Tuesday, January 8, 2013

Trying to create a 1TB database


Looking at transfer speeds to my external hard drive via esata and usb2.
My laptop doesn't support usb 3 or firewire 800


Western Digital My Book Studio Edition II 2TB Desktop External Hard Drive
2 x 1TB 7200rpm drives inside, USB 2.0, eSATA, Firewire400 & 800, RAID 0

Rotational Speed
7200 rpm

http://www.wdc.com/en/products/products.aspx?id=210

http://www.dell.com/us/enterprise/p/latitude-e6520/pd


http://superuser.com/questions/200229/how-can-i-calculate-how-much-time-it-will-take-to-transfer-10-tb-at-different-sp

https://www.google.com/search?q=10TB%2F+1000Mbps

search on google: 1TB/ 100Mbps


dd if=/dev/zero of=/run/media/pvz/4a4f8581-b326-4b06-8910-da03a7a8af68/temp bs=1M count=1024 conv=fdatasync,notrunc
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 10.2618 s, 105 MB/s


1TB/ 100MBps =
   (1 terabyte) / (100 MBps) = 2.91271111 hours


  (1 terabyte) / (100 Mbps) = 23.3016889 hours
More about calculator.


http://www.wolframalpha.com/input/?i=10+terabytes+at+60mbps

http://www.wolframalpha.com/input/?i=1+terabytes+at+105MBps

http://superuser.com/questions/9229/how-long-would-it-take-to-transfer-1tb-over-usb

http://www.calctool.org/CALC/prof/computing/transfer_time

Nice list of transfer technologies and their speeds
http://techinternets.com/copy_calc

http://www.convert-me.com/en/convert/data_transfer_rate/tebibytepday.html


Transfer and write speed using esata:

dd if=/dev/zero of=/run/media/pvz/4a4f8581-b326-4b06-8910-da03a7a8af68/temp bs=1M count=1024 conv=fdatasync,notrunc
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 10.2618 s, 105 MB/s


Transfer and write speed using usb 2:
 dd if=/dev/zero of=/run/media/pvz/4a4f8581-b326-4b06-8910-da03a7a8af68/temp bs=1M count=1024 conv=fdatasync,notrunc
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 39.7351 s, 27.0 MB/s


I am getting about Firewire 800 speeds: 800 (786.432 Mbit/s) based on http://techinternets.com/copy_calc

So between FireWire 800 and Sata 1.

Why I am I not getting Sata 2 or 3 speeds?

Thursday, October 25, 2012

Do we apply Software Engineering principles?

( I don't want to get into the fight about what a software engineer is or what an architect is in this post. We can create one for another day :)

I am wondering if we are really applying software engineering principles in the industry?

I keep telling myself that there must be companies out there that are applying  software engineering principles and good software development practices.

We all assume big companies like Microsoft, IBM, Google, Yahoo, Facebook,etc are hiring good software engineers that are applying good software engineering principles and practices

Then we look at companies like JBoss (Redhat), ThoughtWorks and organisations like Apache and we hope that they are applying software engineering principles and practices.

Maybe we assume that, because they are successful, that they must be applying the knowledge that have been gathered over the years.

Their software engineers must be applying what they learned at university or what they read and learned from good software engineering books.

I hope they are because I haven't experienced it in the last 12 years in the IT industry in South Africa.

I remember what my one lecturer told us in my honours year of computer science: "Remember what you have learned and don't forget what you have learned. Apply what you have learned and fight to apply it. The industry that you are going into is going to ask you to forget it" (Or something to this effect.......it was 11 years ago)

From my experience thus far he is 100% correct.

 Most companies, it seems, are just interested in throwing a few developers together, giving them some requirements and hope they make the deadlines. If they don't succeed then they just try the same thing over and over and over.

What I have experienced so far is this:

  • No effort is placed in employing managers that understand software engineering.
  • No effort is placed in learning from past mistakes.
  • No effort is placed in getting teams to "gel"
  • No effort is placed in handling conflict
  • Scope creep is common and deadlines are not adjusted
  • etc.
I am sure I can go on and on. 
I am sure there are many blogs and wiki's out there that discuss software engineering.

For the next few months this will be the place where I rant and rave about my latest thoughts on the subject.

I have to be honest, I haven't looked at software engineering topics for a while. 
So bear with me.....

For the last few years I was consumed by writing web applications and learning all the cool technologies in the industry :)


Cool technologies are fun.........but for some reason I haven't been very happy the last few years.

I am frustrated with working on projects that miss deadlines or causes people to get so frustrated that they leave.

Yes, the technologies were cool on those projects, and yes, all of them have jumped on the band wagon and called it "Agile Development" and some of them started out using SCRUM..........but yet I am not happy.

Over the last year and a half I wanted to understand what is causing me to be unhappy.

I then made the mistake to read books like these:

  • Peopleware: Productive Projects and Teams by Tom DeMarco and Timothy Lister
  • Death March by Edward Yourdon 
  • The Five Dysfunctions of a Team: A Leadership Fable by Patrick Lencioni 
Of course after reading these books I was energised.....there were people out there researching how people and software development can be combined in perfect harmony......great news.........but......

I also realised I was very sad.

Why are we not applying the knowledge gathered over the years in these books?
When I searched for these books I, of course, found many other books on software engineering/development (mmm....seems I cannot decide on the correct terms).

So why don't I see people applying it in the industry?

Are they scared? Is there too much pressure to deliver? 

Do all the development managers have this knowledge?
Do all the developers have this knowledge and if they do, are they willing to practice it?

So what is wrong in the industry? Is there something wrong or have I just been unfortunate?

I have some theories but I won't share them now. Maybe I will share them over the next few months.....

It has also become clear to me that sometimes if one suggests that one should follow "software engineering principles and practices" that person becomes an outcast. People might look at that person strangely or say yes we are following it or that we are doing  "agile" and yet my frustration is still there..............

I am starting to wonder if I am alone..........maybe I am a lunatic and should take a chill pill and just go with the flow.......

But I cannot do this. The books and my background prevents me from going with the flow. It makes me kick and scream against this..............

I tried forgetting about this subject. I tried reading more technical books like:
  • Clean Code: A Handbook of Agile Software Craftsmanship by Robert C. Martin
  • Effective Java (2nd Edition) by Joshua Bloch
  • Practical API Design: Confessions of a Java Framework Architect by Jaroslav Tulach 
but........for once the technical books didn't help my frustration....and....I found them sometimes saying you should do X and then the other book saying you shouldn't do X.  Plus I found people misreading Clean Code and not allowing any comment in the code.

While the books above is great and technical it still didn't solve my frustration in the industry on projects.

I really wanted to understand my frustration.

I wanted to work in a company that valued Peopleware. I wanted to create such an environment.  

But after much prompting and suggestions I am still frustrated.

So what to do?

I have decided to write a few blogs on the topic of software engineering and vent my frustration.

I am sure it has been done before so if you are not really into another one you can safely move on :)

I have decided that I will use the book that I am currently reading as my guide or template for my venting: Rapid Development: Taming Wild Software Schedules by Steve McConnell

So in the coming months I will blog about the chapters that interest me and use the chapters to present my views and experiences on them.

If you are interested and keen tune in next week for the start of my structured and focused venting :)

PS: Yes...I am more into Java, databases and open source and will still one day write my own database engine.......but for now let me vent on software engineering.
Also if you have any good blogs or web sites to share on this topic, please do.

Disclaimer: I have worked with good project mangers, software development managers and architects.........but they have only been a handful...and they are the exception rather than the rule.