User story (meta) #1988

When the postgresql database is restarted, the application is not usable anymore

Added by Nicolas CHARLES over 1 year ago. Updated about 1 year ago.

Status:Released Start date:2011-11-03
Priority:4 Due date:
Assignee:François ARMAND % Done:

100%
Category:Architecture - Refactoring
Target version:2.4.0~alpha7
Needs translating:No

Description

It seems the connctionPool is not trying to invalidate old connection and create new ones

The stacktrace shows :

15:51:04.423 [pool-1-thread-14] ERROR net.liftweb.actor.ActorLogger - Actor threw an exception
org.springframework.dao.DataAccessResourceFailureException: PreparedStatementCallback; SQL []; FATAL: terminating connection due to administrator command; nested exception is org.postgresql.util.PSQLException: FATAL: terminating connection due to administrator command
at org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:104) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:602) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:842) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at com.normation.rudder.repository.jdbc.EventLogJdbcRepository.saveEventLog(EventLogJdbcRepository.scala:84) ~[EventLogJdbcRepository.class:na]
at com.normation.rudder.services.log.EventLogServiceImpl.saveEventLog(EventLogServiceImpl.scala:52) ~[EventLogServiceImpl.class:na]
at com.normation.rudder.batch.AsyncDeploymentAgent$$anonfun$lowPriority$1.apply(AsyncDeploymentAgent.scala:124) ~[AsyncDeploymentAgent$$anonfun$lowPriority$1.class:na]
at com.normation.rudder.batch.AsyncDeploymentAgent$$anonfun$lowPriority$1.apply(AsyncDeploymentAgent.scala:112) ~[AsyncDeploymentAgent$$anonfun$lowPriority$1.class:na]
at scala.PartialFunction$$anon$1.apply(PartialFunction.scala:46) ~[na:na]
at net.liftweb.actor.LiftActor$class.execTranslate(LiftActor.scala:441) ~[lift-actor_2.9.0-1-2.4-M3.1.jar:2.4-M3.1]
at com.normation.rudder.batch.AsyncDeploymentAgent.execTranslate(AsyncDeploymentAgent.scala:83) [AsyncDeploymentAgent.class:na]
at net.liftweb.actor.SpecializedLiftActor$class.net$liftweb$actor$SpecializedLiftActor$$proc2(LiftActor.scala:289) [lift-actor_2.9.0-1-2.4-M3.1.jar:2.4-M3.1]
at net.liftweb.actor.SpecializedLiftActor$$anonfun$net$liftweb$actor$SpecializedLiftActor$$processMailbox$1.apply(LiftActor.scala:210) [lift-actor_2.9.0-1-2.4-M3.1.jar:2.4-M3.1]
at net.liftweb.actor.SpecializedLiftActor$$anonfun$net$liftweb$actor$SpecializedLiftActor$$processMailbox$1.apply(LiftActor.scala:210) [lift-actor_2.9.0-1-2.4-M3.1.jar:2.4-M3.1]
at net.liftweb.actor.SpecializedLiftActor$class.around(LiftActor.scala:224) [lift-actor_2.9.0-1-2.4-M3.1.jar:2.4-M3.1]
at com.normation.rudder.batch.AsyncDeploymentAgent.around(AsyncDeploymentAgent.scala:83) [AsyncDeploymentAgent.class:na]
at net.liftweb.actor.SpecializedLiftActor$class.net$liftweb$actor$SpecializedLiftActor$$processMailbox(LiftActor.scala:209) [lift-actor_2.9.0-1-2.4-M3.1.jar:2.4-M3.1]
at net.liftweb.actor.SpecializedLiftActor$$anonfun$2$$anonfun$apply$mcV$sp$1.apply$mcV$sp(LiftActor.scala:173) [lift-actor_2.9.0-1-2.4-M3.1.jar:2.4-M3.1]
at net.liftweb.actor.LAScheduler$$anonfun$9$$anon$2$$anon$3.run(LiftActor.scala:64) [lift-actor_2.9.0-1-2.4-M3.1.jar:2.4-M3.1]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [na:1.6.0_25]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [na:1.6.0_25]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_25]
Caused by: org.postgresql.util.PSQLException: FATAL: terminating connection due to administrator command
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2077) ~[postgresql-8.4-702.jdbc4.jar:na]
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1810) ~[postgresql-8.4-702.jdbc4.jar:na]
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257) ~[postgresql-8.4-702.jdbc4.jar:na]
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:498) ~[postgresql-8.4-702.jdbc4.jar:na]
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:386) ~[postgresql-8.4-702.jdbc4.jar:na]
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:332) ~[postgresql-8.4-702.jdbc4.jar:na]
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) ~[commons-dbcp-1.4.jar:1.4]
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) ~[commons-dbcp-1.4.jar:1.4]
at org.springframework.jdbc.core.JdbcTemplate$3.doInPreparedStatement(JdbcTemplate.java:844) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate$3.doInPreparedStatement(JdbcTemplate.java:1) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:586) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
... 19 common frames omitted

Associated revisions

Revision 4959b117
Added by François ARMAND about 1 year ago

Fixes #1988: make postgres connection be check for death before being used

Revision 48cff233
Added by Nicolas CHARLES about 1 year ago

Fixes #1988 : change the query to validate the connectivity to be compliant with new databases

History

#1 Updated by Jonathan CLARKE over 1 year ago

  • Target version set to 2.4.0~alpha3

#2 Updated by Jonathan CLARKE over 1 year ago

  • Target version changed from 2.4.0~alpha3 to 2.4.0~alpha4

#3 Updated by Jonathan CLARKE over 1 year ago

  • Priority changed from N/A to 4

#4 Updated by François ARMAND over 1 year ago

  • Target version changed from 2.4.0~alpha4 to 2.4.0~alpha5

#5 Updated by Jonathan CLARKE over 1 year ago

  • Target version changed from 2.4.0~alpha5 to 2.4.0~alpha6

#6 Updated by Jonathan CLARKE about 1 year ago

  • Target version changed from 2.4.0~alpha6 to 2.4.0~alpha7

#7 Updated by François ARMAND about 1 year ago

  • Status changed from New to In progress
  • Assignee set to François ARMAND

The first step is to at least gracefully handle JDBC exception.

#8 Updated by François ARMAND about 1 year ago

Hum, there is too many jdbcTemplate call to wrap them all now... That will lead to surprises, so that will tracked here: #2438

#10 Updated by François ARMAND about 1 year ago

Oups,bad pooling driver: http://commons.apache.org/dbcp/configuration.html see validationQuery

#11 Updated by François ARMAND about 1 year ago

  • Status changed from In progress to Pending technical review
  • % Done changed from 0 to 100

#12 Updated by Nicolas CHARLES about 1 year ago

  • Status changed from Pending technical review to Released

This looks highly valid, thank you Francois

#13 Updated by Jonathan CLARKE about 1 year ago

  • Tracker changed from Bug to User story (meta)
  • Status changed from Released to Pending functional review

I would like to validate this behaviour on a production-like environment, to see how it should be documented and explained - so changing to a user story (it sort of is - "Jim shouldn't have to call Normation to get Rudder to work after restarting his PSQL server" :) ) to add a functional review step.

#14 Updated by Nicolas CHARLES about 1 year ago

  • Status changed from Pending functional review to In progress
  • % Done changed from 100 to 50

Actually, this modification breaks at the initial startup of the application (when the database is empty), for there are no entries in the eventlog database

#15 Updated by Nicolas CHARLES about 1 year ago

I suggest this query :
SELECT tables.table_name FROM information_schema.tables WHERE lower(table_name) = 'eventlog';

#16 Updated by Nicolas CHARLES about 1 year ago

  • Status changed from In progress to Pending technical review
  • % Done changed from 50 to 100

#17 Updated by Nicolas CHARLES about 1 year ago

  • Assignee changed from François ARMAND to Nicolas CHARLES

#18 Updated by François ARMAND about 1 year ago

  • Status changed from Pending technical review to Pending functional review

The proposed modification is ok

#19 Updated by Jonathan CLARKE about 1 year ago

OK, this works when the DB is stopped then restarted. However, error handling when the DB is unavailable is still non functional.

#20 Updated by Jonathan CLARKE about 1 year ago

  • Status changed from Pending functional review to Released
  • Assignee changed from Nicolas CHARLES to François ARMAND

Oops, forgot to actually close this one.

Also available in: Atom PDF