Few notes on our experience while preparing for baseline

Few notes on our experience while preparing for baseline

by Jessica Aguti -
Number of replies: 7

Dear Colleagues,

It was great seeing some of you yesterday in  the Zoom meeting we had with Betty and May.  Now, in the meeting, we agreed that I share with us all some of the issues we grappled with as we prepared for the baseline and as went through the instruments.

  1. There are a number of instruments and we found these generally okay and not problematic.  However this was different with the Environment Awareness instrument for student.   Because this study involves many countries, changes to the instruments have to be minor so we do not have to go through the validation processes afresh.  So, the changes made to the Environment Awareness instrument were minor.  We made the instrument relevant to our context without compromising quality because we felt some of the items in the generic document were either not relevant to us in Uganda or likely not to be understood by the students.
  2. We also decided that we needed an instrument that captures data for a school profile.  We think this could help us appreciate/understand certain practices in the school.  This not one of the generic instruments distributed by COL.  Even though this data might not necessarily be shared with COL, we concluded that understanding the schools we are working with would be informative in the later stages of the project.  That particular instrument is attached here for your information.
  3. The lesson observation instrument is another that gave us some pain especially with regard to timing of the different lesson activities.  Enumerators felt a lot of time was going to be spent looking at the watch and not observing the lesson.  WE therefore reformatted this instrument in the manner the enumerators felt would work best for them.  I am yet to establish how specifically this worked out in the field.  Attached is this reformatted instrument
  4. The same instruments were used for the control schools  
  5. We organize the training of the enumerators close to the dates for fieldwork so as to ensure whatever is learnt during the training is not forgotten before fieldwork takes place.

Sorry for the long post.  But I hope someone out there will find this helpful

Take care and stay safe


In reply to Jessica Aguti

Re: Few notes on our experience while preparing for baseline

by Jessica Aguti -

Guys 

Everyone is quiet.  I hope you are all well even in the midst of COVID 19.  Please take care 

In reply to Jessica Aguti

Re: Few notes on our experience while preparing for baseline

by Alhassan Hamza -

Hope everyone is fine in this period of COVID 19. Fortunately for us, we had our baseline survey before the COVID 19.

Best regards.

Hamza Alhassan

TACE, GHANA

In reply to Jessica Aguti

Re: Few notes on our experience while preparing for baseline

by Prof. Robert McCormick -

Dear Jessica,

Thanks for those informative points and flags for what to think about, which others who have yet to do a baseline study can learn from.

The point you make on the need to localise the Environmental Awareness questionnaire is really important, to make it relevant to the local environment and environmental concerns. As I remember you were able to draw upon a local environmental expert to help with this task, and this is also an useful approach.

On the school profile, it is good to see you introducing your own instrument to help with your work with the schools subsequently. Your colleague mentioned that the analysis was underway in early June and I wondered if it had yet revealed any lessons that will be important for the implementation of TFP.

The point you make about the Classroom Observation Schedule is also an important one.

I can imagine that the CPO schedule is somewhat daunting, That is why in the enumerator training I advised that videos of real local classrooms should be used as part of the training, though I understand that this is not always possible. Only practice and discussion of coding in detail (i.e. first trying to code live while observing, then stepping through the video at the 4—minute intervals and checking what everyone codes at each point, with discussion of deviations) enables consistent and reliable coding to take place. The fact that the coding is instantaneous means that it is not necessary to try to summarise the 4-minute period in the code, but just say what is happening at the time of the 4-minute interval. (I have modified the Enumerator Handbook to emphasise this point as a result of your comments.)

The two other points you make on using the same instruments in control schools as in treatment schools, and on having the enumerator training close the data collection time, are also two useful things to emphasise. 

it would be interesting to hear from others who have done a baseline study or who are thinking about it, to broaden the concerns. Thanks for kicking this discussion off.

Best wishes

Bob



In reply to Prof. Robert McCormick

Re: Few notes on our experience while preparing for baseline

by Prof. Robert McCormick -

Dear All, and especially Jessica

I have no idea how I orinially put Betty's name in addressing the above comments. I was sure I wrote 'Dear Jessica', who of course made the comments in the message I was replying to. My sincere apologies to Jessica for the mistake and my thanks to her for the comments she made.

Thanks to Betty for making the change to my first reply to Jessica.

Best wishes


Bob 

In reply to Prof. Robert McCormick

Re: Few notes on our experience while preparing for baseline

by Betty Ogange -

Dear Bob,

Many thanks for the insightful comments. I have used my 'privilege' to correct the addressee on your post just for accurate reference and communication in future. Jessica and yourself have had very valuable discussions on the Uganda baseline that  I hope our colleagues will find helpful going forward.

Considering the changes coming about in the schooling systems in most countries as a result of the COVID-19 crisis, it is apparent that we will be re-considering the kind of questions we ask in the baselines. What, for instance, would change in the classroom observation tool if teaching and learning took place in a learning management system, where there are both synchronous and asynchronous learning experiences? 

It would be great to get some insights from colleagues in this regard. Colleagues may respond by providing brief answers to the following TWO questions:

  1. What are some of the key changes currently taking place in the teacher training and schooling systems in your country?
  2. What would you like us to pay attention to as we plan the baselines and the broader research, monitoring and evaluation strategies for the immediate future?
For ease of engagement, we request colleagues to keep their responses on this thread.

Best regards,

Betty


In reply to Betty Ogange

Re: Few notes on our experience while preparing for baseline

by Alpha Bah -

In response to Q1 the key changes in the education response strategy to Covid-19 is safe reopening of schools with an underlying requirement of observing social distancing. In The Gambia: reopening is done In phases and class size is reduced to 20 to 25 students. Seating arrangements is like during examination period where you have each student in a desk. Hence conventionally  learner centered and student collaboration and engagement without the use of technology is somewhat limited. 

As regards Q2,

Going forward with the baseline and the broader M&E it would be useful to orient the design towards both the learner and the tutor’s socioeconomic environment; lessons from COVID-19 and the disruption of school based learning: education system lost control of the distance education that was being provided as a mitigation and continuing of learning - there was very limited feedback and knowledge of the learning environment. Therefore, the need to position the evaluation for an individual learner case cannot be overemphasized 

In reply to Alpha Bah

Re: Few notes on our experience while preparing for baseline

by Prof. Robert McCormick -

Dear Alpha

I waited to see if those of our colleagues who are working in the field would respond, but here are a few thoughts meanwhile.

The first thing to say is that the M&E strategy, and hence the baseline, were designed in 2018, a long time before we moved into this unusual situation where the learning environment has changed so much. That makes it somewhat unsuited to those of you who are now dealing with a completely different situation, perhaps where instead of schools, small groups are being brought together using local community resources such as volunteers to lead these groups; or perhaps even only phone calls to isolated students who have either no local group or are working in informal household groups. Those fortunate to have online access might be doing more 'traditional' distance learning. 

Thus any programme that is supporting teachers (and volunteers) during this has to be quite different. WhatsApp groups have been popular and well 'attended' by teachers, where training can be used. If material has already been made available to teachers, then they can work through this and meet each week in a semi-synchronous way and exchange text messages. Photographs can be exchanged to show what learning situations look like or work that students have produced. Audio can also be used.

Designing an evaluation for this is quite different, with more reliance on using the digital record (it is possible to download the WhatsApp text; there is software on the web to feed this into, to produce aggregate statistics (like number on the session, and who posts, when). it is also possible to look at the text and to analyse what is said, but that has to be done by individuals and is not straightforward. All this give some monitoring information. 

To get more information requires say telephone interviews with contributors and those who participate but don't post. These interviews can be simple questionnaire responses about how participants feel about the sessions (e.g. how useful, difficult to follow, inconvenient), or can ask about how the ideas discussed have helped them to cope with the teaching and learning they are involved with in the new situations I have described above.

Most of the above count as monitoring activity, but at this stage it is not unimportant. If proper sampling and rigorous instruments are used, it could be used to evaluate the activity (e.g. by interviewing participants more than one time). 

You can see that even with this is one example (use of WhatsApp) that the evaluation will look nothing like the one designed for the TFP in 2018! If we also include online web activity for teacher training and support (which many of you might already have done or are developing), then the evaluation and monitoring requirements would be different again.

Even when students return to schools, social distance may change the situation and the training and support for teaching has to adapt.

As I understand it, the Commonwealth of Learning is trying to respond to the current and future situation to support its partners. Perhaps Betty can give us a hint of what might be to come (no pressure Betty!)?

Best wishes

Bob